Debugging tricky parameterized types

Parameterized functions are fantastic. I'm talking about the functions that operate around some abstract piece of data, without doing anything that would require specific knowledge about that data. Take, for example, the following fictional function:

Example

let maybeSaveThing = (maybeThing: option<debug>): option<debug> => {
  switch maybeThing {
  | Some(thing) => saveThing(thing)
  | None => None
  }
}

The code above is designed to take an option of something, save it somewhere if the option contains data, and return the result. We assume, for this example, that the saveThing function knows now to save anything. We can tell that the function shouldn't care what's contained in the option type by looking at the function signature. It takes an option<'a>, and returns an option<'a>.

A Sneaky Bug

But this code has a sneaky bug. We can see it if we look at the function's inferred type:

option<int> => option<int>

The compiler is telling us that the function takes an option of an integer, even though we explicitly stated in the type signature that the function should be generic. Why is that?

Why Don't we Get a Type Error?

Generics encapsulate all other types, that's why. For example, an option<int> is an option<'a> as far as the type system is concerned. So is an option<string> or a option<blah>. If a part of the code inside of a function returns a specific value where a generic value was expected, the type inference algorithm quietly turns the function's parameter into a specific type. The compiler won't tell you that your type annotation is wrong, because it isn't. An option<int> => option<int> is an option<'a> => option<'a>.

Why Are We Getting an int?

Spoiler alert, there's a call inside of our function that returns an int where the generic type was expected. Stick around and we'll see how to find it in a moment.

Discovering the Mismatch

We intended the type to be generic. It's troublesome that we don't discover this breakage until we want to use this function to save some type other than an int. The compiler will tell us that maybeSaveThing takes an option<int> and we're trying to pass in something like option<string>.

How can we discover these unintentional losses of genericism closer to the site of definition?

Enforcing Genericism

Adding an interface (.resi) file with the generic signature for the function will help the compiler out. Instead of inferring the type of the function to be specific, the compiler will know that you intend to keep that function generic, and it'll tell you that your implementation doesn't match the interface.

Finding the Mistake

Unfortunately, an error that the function definitions don't match doesn't go far to help us find the location of the mistake. When we're dealing with more than mere example code, and we have a complex function with lots of operations, it's tempting to abort the whole effort and scroll through Twitter instead.

But here's a trick that can speed the process up:

type debug

We've added a new type to our code that is the opposite of generic. It's so specific that no value fits it. And it's explicitly for debugging, so we know that no other area of the code will rely on this type for business logic.

If we swap out the generic argument in our function definition with debug temporarily:

type debug

let maybeSaveThing = (maybeThing: option<debug>): option<debug> => {
  switch maybeThing {
  | Some(thing) => saveThing(thing)
  | None => None
  }
}

It'll show us the line of code that is returning a specific, non-generic value:

  8 │ let maybeSaveThing = (maybeThing: option<debug>): option<debug> => {
   9 │   switch maybeThing {
  10 │   | Some(thing) => saveThing(thing)
  11 │   | None => None
  12 │   }

  This has type: option<int>
  Somewhere wanted: option<debug>

  The incompatible parts:
    int vs debug

Aha! The call to saveThing is returning a option<int>. If that had been a generic function, its return type would have been option<debug>. Let's look at its definition:

let saveThing = _thing => {
  Some(21)
}

There's the source of the problem. If we modify it to do something generic instead:

let saveThing = thing => {
  let _ = Js.Json.stringifyAny //-> blah blah do some saving in the background
  Some(thing)
}

Then the compiler is happy, and we know our code is generic again. We can remove debug and put the type parameters back in place.

On File Switching in Rescript

I use Rescript at work, and find that I often want to switch between interface and implementation files. Less often, I also want to inspect the compiled source of a file. This can be a small pain to do by hand, and small pains repeated often become big pains. So here are some VS Code extensions to help out.

For Opening Interface & Compiled Files

Extension: Open related files

This extension will automatically open files with the same name, but different extensions, so you'll get the .resi and the .bs.js files popped up in new tabs. Better yet, if you keep running the command, it toggles between .res, and .resi.

This one doesn't come with a default keymap, so you'll need to add one in your settings.

Another cool thing about this extension is that it can create interface files for you as well, just throw this into your settings.json:

  "openRelatedFiles.createFileMap" : {
    ".res": [
        ".resi"
    ],
    ".re": [
      ".rei"
    ]
  }

And then you can use the "Create related files" command to make an ".resi" file for you.

For Opening Child Files

The OCaml naming convention gives files hierarchy by their names. For example Foo.res would break out some complex logic "baz" into a subfile: Foo_Baz.res and import it. We can use an extension to easily browse these child files.

Quick Open Related Files

This pops open a picker of files that have a name with a prefix similar to our own:



Animating Text Content with Redash and Reanimated 2

On Cyber Monday I sprang for William Candillon's course on animation with React Native. I love his style. And I like how beautiful his examples are. However, it took me a few minutes to understand that he's basing all his work on two libraries, Redash, and Reanimated. These tools are super powerful, and I'm off to a strong start building lovely app animations, but I ran into a couple of problems while getting started.

  1. TypeError: global.__reanimatedWorkletInit is not a function

I got this error the first time I tried to use Redash. I had installed both it and reanimated, imported a hook, and tried to use it. But the app immediately crashed. I resolved it by importing reanimated at the top of the file:

import "react-native-reanimated";
import ...
  1. Animated string values would cause a rerender in the UI.

With useDerivedValue I transitioned a string into existence by animating its length. However, when I passed the Animation.SharedValue<string> to a Retext component and tested it out in my app, there was no animation.

Thanks to an answer this github issue I fixed that problem by whitelisting text as an animatable prop:

Animated.addWhitelistedNativeProps({ text: true });

After adding that line to the top of the file, my views updated without a problem as the text animated.

On media uploads, and annoying S3 APIs.

While working on Storytime Studio a couple of weeks ago I decided to upgrade my media uploading approach. I had started with the naive "I'll just upload everything to the server and put it in a folder on the hard drive" approach, because that was simple. And it worked really well until I tried to upload 15 minutes of audio (a few hundred megabytes) at once. The server crashed, every time. Even after upgrading the box from 1GB of memory to 2GB.

The server is written in Adonis JS right now (an experiment, I've never used that framework at all before, let alone in production) and it's taking care of parsing the multipart form body I was using. I assume that the body parser is streaming the content to a temporary file on disk. It shouldn't be a big memory hit, even with a file that's hundreds of megabytes. But instead of taking time to figure out why the server wasn't surviving even a moderately heavy upload, I decided it was time to switch to the direct-to-s3 upload model that I would eventually be moving to anyway.

Before I move on, let me make sure you know what S3 is. Amazon's S3 is a low-cost, durable option for storing even very large files indefinitely. It's a fantastic place to put media files and raw data. I'm using S3 as a loose term here, because I'm actually hosting my server on Linode, and I'll be using Linode's Object Storage, which implements the same API as S3. Most cloud platforms have some service like this.

What is the direct-to-S3 approach?

Short and sweet: The server generates a special URL for putting data directly in S3 and hands that back to the client. The client can use that URL to upload or download media for a limited time. After that time, the URL expires. The media itself doesn't have to go through the server, and since the URLs are time-limited, it's secure enough for most purposes.

The requests look like this when uploading audio:

sequenceDiagram participant Phone participant Server participant S3 Phone->>Server: Upload story meta Server->>Phone: Signed URL Phone->>S3: Upload story audio file

The audio content never actually touches my server, only the metadata. The audio goes straight to S3. This pattern drastically increases the amount of traffic my little application server will be able to handle, because it doesn't have to worry about buffering, writing or reading any of the media data coming from the clients. And even a small server can handle a considerable number of requests for little chunks of metadata. Thus, this is a "scalable" way to handle media upload.

This is what the requests look like for downloads:

sequenceDiagram participant Phone participant Server participant S3 Phone->>Server: Get story content Server->>Phone: Redirect to signed URL Phone->>S3: Follow redirect and download content

When the phone asks for story content from the server, the server just sends back a 302 Found status code, and sets the Location header to the newly generated S3 URL. The client then automatically visits that URL and downloads the content. Again, nothing has streamed through my little inexpensive server, and my scalability goes up.

API Pain

The current version of Amazon's S3 API was designed in 2006, and it shows its age. Though doing simple things isn't too uncomfortable, and there's not much drama if you don't make any mistakes, I've found debugging mistakes when they do happen to be a real pain. Switching to this direct-to-s3 approach took me multiple days, when it should have taken me an hour or two, because I was busy debugging APIs that didn't feel like they should have been breaking.

Initially I used the "aws-sdk" node module to generate signed URLs for Linode's object storage. I do this all the time at Day One (where we are using the real S3), and I've never had a problem with it. But no matter what I tried, I kept getting a response from Linode that said SignatureDoesNotMatch, and that's pretty much all it said 😡. Yeah, it seems they use that error to cover a number of possible mess-ups. So it took me a lot of experimentation to finally get anything working.

I ditched the aws-sdk module and followed their docs for their HTTP API instead. These docs taught me two important things:

  • When using Linode's object storage, the content-type of the media being PUT to the URL must be specified when creating the URL, and the upload must match that type.
  • When doing a GET or a DELETE, the content-type must not be specified when creating the URL.

Guess what happens if you mess one of those things up? SignatureDoesNotMatch. Ultimately that error message does make sense, once you understand that the content-type is part of the signature, but the information returned in the response is bad at actually helping the user to fix the problem.

Even though I ditched the S3 API library, I think now that I probably could have stuck with it if I had included and excluded the content-type attribute in the right places.

Peace

It's done now. I probably won't have to touch this part of the code again for a very long time, if ever. So even though the work was frustrating, I can close my eyes, take a deep breath in, let it out, and move on to other frustrating problems 🧘‍♂️.