Bifröst - a queryable in-process worker queue (golang)

Aug 24, 2017 20:00 · 421 words · 2 minutes read

robo

Tl;DR

I wrote a simple in-memory job queue for golang. All it does is spin up a set of background worker goroutines, then accepts sending anything that implements interface type JobRunner interface {Run() error}) to the workers as jobs.

Other features:

Query jobs based on job ID
Returned ‘tracker’ that can be used to check status or wait on completion
Job status is JSON serializable, REST API friendly

Check out the project on Github: Bifröst - Golang query-able job queue.

Example usage:

package main

import (
    "encoding/json"
    "fmt"
    "github.com/serdmanczyk/bifrost"
    "os"
    "time"
)

func main() {
    stdoutWriter := json.NewEncoder(os.Stdout)
    dispatcher := bifrost.NewWorkerDispatcher(
        bifrost.Workers(4),
        bifrost.JobExpiry(time.Millisecond),
    )

    // Queue a job func
    tracker := dispatcher.QueueFunc(func() error {
        time.Sleep(time.Microsecond)
        return nil
    })

    // Queue a 'JobRunner'
    dispatcher.Queue(bifrost.JobRunnerFunc(func() error {
        time.Sleep(time.Microsecond)
        return nil
    }))

    // Print out incomplete status
    status := tracker.Status()
    stdoutWriter.Encode(&status)
    // {"ID":0,"Complete":false,"Start":"2017-03-23T21:51:27.140681968-07:00"}

    // wait on completion
    <-tracker.Done()
    // Status is now complete
    status = tracker.Status()
    stdoutWriter.Encode(&status)
    // {"ID":0,"Complete":true,"Success":true,"Start":"2017-03-23T21:51:27.140681968-07:00","Finish":"2017-03-23T21:51:27.140830827-07:00"}

I was working on Freyr, and decided I wanted to implement the ability to post a list of data.

I had a few options for implementing this

Just add a REST handler, perform all DB inserts before returning
As above, but perform inserts in a goroutine
Implement a query-able worker queue of limited length, delegate DB operations to queue

The solutions go from simple->complex as well as worst->best.

Just performing the inserts during the REST call can lead to painfully long call waits with long lists of data. (a years worth of readings took 40 seconds to insert one-by-one).

Delegating to goroutines can quickly overload the database with several insert operations going on at once.

A job queue would limit the number of goroutines, as well as allow a client to query status.

Inspiration

The job queue-ing method is nothing new. Blog posts already exist detailing methods for writing one.

What I did was wrap the method detailed in those posts and add the features mentioned in the TD;DR at the beginning of the post.

The Result

So naturally, to add a single feature to one project I wrote an entire separate project. Albeit a small one.

Hopefully this provides something interesting for others to look at or use in their hobby projects. Production libraries will likely want more features such as the ability to horizontaly scale the queue i.e. across processes/machines.

Link again: Bifröst - Golang query-able job queue.

Tl;DR

When you give a mouse a cookie, he writes a job queue

Inspiration

The Result