diziet | derive-adhoc: powerful pattern-based derive macros for Rust (Reply)

tl;dr

Have you ever wished that you could that could write a new derive macro without having to mess with procedural macros?

Now you can!

derive-adhoc lets you write a #[derive] macro, using a template syntax which looks a lot like macro_rules!.

It’s still 0.x - so unstable, and maybe with sharp edges. We want feedback!

And, the documentation is still very terse. It is doesn’t omit anything, but, it is severely lacking in examples, motivation, and so on. It will suit readers who enjoy dense reference material.

Background - Rust’s two (main) macro systems

Rust has two principal macro systems.

macro_rules! (also known as “macros by example”) is relatively straightforward to use. You have some control over the argument syntax for your macro, and then you can generate output code using a pattern-style template.

But, its power is limited. In particular, although you can specify a pattern to match the arguments to your macro, the pattern matching system has serious limitations (for example, it has a very hard time with Rust’s generic type parameters). Also, you can’t feed existing pieces of your program to a macro without passing them as arguments: so you must write them out twice, or have the macro re-generate its own arguments as a side-effect.

Because of these limitations, code which makes heavy use of macro_rules! macros can be less than clear.

Procedural macros are an extremely powerful and almost fully general code-rewriting macro facility. They work by taking actual rust code (represented as a stream of tokens) as their input, running arbitrary computations, and then generating a new stream of tokens as an output. Procedural macros can be applied (with derive) to Rust’s data structure definitions, to parse them, and autogenerate data-structure-dependent code. They are the basis of many extremely powerful facilities, both in standard Rust, and in popular Rust libraries.

However, procedural macros are hard to write. You must deal with libraries for parsing Rust source code out of tokens. You must generate compile errors manually. You often end up matching on Rust syntax in excruciating detail. Procedural macros run in an inconvient execution context and must live in a separate Rust package. And so on.

“Derive by example” with `derive-adhoc`

derive-adhoc aims to provide much of the power of proc macros, with the convenience of macro_rules!.

You write a template which is expanded for a data structure (for a struct, say). derive-adhoc takes care of parsing the struct, and gives you convenient expansion variables for use in your template.

A simple example - deriving `Clone` without inferred trait bounds

Here is a simple example, taken from clone.rs, in derive-adhoc’s test suite; clone.expanded.rs shows the result of the macro expansion.

This showcases very few of derive-adhoc’s features, but it gives a flavour.

  // Very simple `Clone`
  //
  // Useful because it doesn't infer Clone bounds on generic type
  // parameters, like std's derive of Clone does.  Instead, it
  // unconditionally attempts to implement Clone.
  //
  // Only works on `struct { }` structs.
  //
  // (This does a small subset of what the educe crate's `Clone` does.)
  define_derive_adhoc!{
      MyClone =

      impl<$tgens> Clone for $ttype {
          fn clone(&self) -> Self {
              Self {
                  $(
                      $fname: self.$fname.clone(),
                  )
              }
          }
      }
  }

  // If we were to `#[derive(Clone)]`, DecoratedError<io::Error> wouldn't
  // be Clone, because io::Error isn't, even though the Arc means we can clone.
  #[derive(Adhoc)]
  #[derive_adhoc(MyClone)]
  struct DecoratedError<E> {
      context: String,
      error: Arc<E>,
  }

Replacing an existing bespoke proc macro - a more complex example

Recently, I thought I would try out derive-adhoc in Hippotat, a personal project of mine, which currently uses a project-specific proc macro. This was an enjoyable experience.

I found the new code a huge improvement over the old code. I intend to tidy up this branch and merge it into Hippotat’s mainline, at some suitable point in the release cycles of Hippotat and derive-adhoc.

I won’t copy the whole thing here, but: now we have things like this:

  pub struct InstanceConfig {
    #[adhoc(special="link", skl="SKL::None")]  pub link: LinkName,
  ...

  derive_adhoc!{
    InstanceConfig:

    fn resolve_instance(rctx: &ResolveContext) -> InstanceConfig {
      InstanceConfig {
        $(
          $fname: rctx.
            ${if fmeta(special) {
              ${paste special_ ${fmeta(special)}}
            } else {

Instead of this kind of awfulness:

   } else if attr.path == parse_quote!{ special } {
     let meta = match attr.parse_meta().unwrap() {
       Meta::List(list) => list,
       _ => panic!(),
     };
     let (tmethod, tskl) = meta.nested.iter().collect_tuple().unwrap();
     fn get_path(meta: &NestedMeta) -> TokenStream {
       match meta {
         NestedMeta::Meta(Meta::Path(ref path)) => path.to_token_stream(),
         _ => panic!(),
       }
     }
     method = get_path(tmethod);
     *skl.borrow_mut() = Some(get_path(tskl));
   }

History and acknowledgements

derive-adhoc was my project proposal in last year’s Tor Project Hackweek. Thanks to Tor for giving us the space to do something like this.

Nick Mathewson joined in, and has made important code contributions, also given invaluable opinions and feedback. Thanks very much to Nick - I look forward to working on this more with you. I take responsibility for all bugs, mistakes, and misjudgements of taste.

Future plans

We’re hoping derive-adhoc will become a widely-used library, significantly improving Rust’s expressive power at the same time as improving the clarity of macro-using programs.

We would like to see the wider Rust community experiment with it and give us feedback. Who knows? Maybe it will someday inspire a “Derive By Example” feature in standard Rust.

Two words of warning, though:

The documentation is currently very terse. Many readers will find it far too dense and dry for easy comprehension. The examples are sparse, not well integrated, and not very well explained. We will need your patience - and and your help - as we try to improve it.
The library and its template syntax are still unstable. As more people try derive_adhoc, we expect to find areas where the syntax and behavior need to improve. While still at 0.x, we’ll be keen to make those improvements, without much regard to backward compatibility. So, for now, expect breaking changes.

We hope to release a more-stable and better-documented version 1.x later this lear.

So, please try it out and let us know what you think.