[repo] Support common query functionality

[benmai]

Follow-up of this Slack conversation.

In the process of working on a new repo driver, I’ve run up against some of the limits of the ReadRepo interface. Namely:

1. Each implementation of it so far in eventhorizon has its own ad-hoc methods. For example, the Mongo driver has FindCustom and FindCustomIter, which take Mongo-specific arguments. The problems with this are two-fold:
   1. It’s difficult to swap out implementations, which makes testing hard (e.g. if we don't have access to a real database or just don't want to spin one up for speed reasons and would rather use local).
   2. The acceptance tests don’t cover a large portion of the ReadRepo functionality, which makes it hard to implement new drivers that follow the conventions of existing drivers and are tested with the same level of rigor.
2. Find and FindAll alone don’t fulfill the query needs of a fully-featured application. It’s missing paging (a must-have for a large number of records), selecting by any kind of condition, making aggregation queries (e.g. count of records which satisfy a condition), sorting, and probably some other things I'm not thinking of. The MongoDB repo solves this in an ad-hoc way by exposing the ability to execute MongoDB queries directly. Also, from what I can surmise the Parent() method on ReadRepo exists only to access this MongoDB-specific functionality at the moment.
3. The interface itself, with its Find and FindAll methods, seems more informed by MongoDB than CQRS. This is a completely semantic argument, but to me naming something after the MongoDB API makes it marginally more difficult to map the functionality to CQRS principles.

Based on these limitations, I would like to propose the following replacement for ReadRepo:

// Querier exposes an interface for retrieiving Entities based on a query.
type Querier interface {
        Query(ctx context.Context, opts ...QueryOption) (*QueryResults, error)
}
    
// QueryOption modifies a query to return limited results based on some
// restriction. A query may take multiple QueryOptions or none.
type QueryOption interface{}
    
// QueryResults represents the result of a query.
type QueryResults struct {
        Entities []Entity // One page of entities matching the query.
        Total    uint64   // The total number of entites matching the query.
}
    
// Possible general QueryOptions.
    
// NewQueryOptionEntityID returns a QueryOption which restricts the results
// of Query to return only a record matching the id (if it exists).
func NewQueryOptionEntityID(id UUID) QueryOption {
        return QueryOptionEntityID{id}
}
    
type QueryOptionEntityID struct{
        id UUID
}
    
// NewQueryOptionSearchTerm returns a QueryOption which when passed to Query
// will perform a full text search over the specified fields using term as
// the search term.
func NewQueryOptionSearchTerm(term string, fields []string) QueryOption {
        return QueryOptionSearchBy{term}
}
    
type QueryOptionSearchTerm struct{
        term   string
        fields []string
}
    
// NewQueryOptionFieldExactMatch returns a QueryOption which restricts query results
// to ones for which the value of the specified field matches exactly one of the
// specified values.
//
// E.g., A query over the Entity collection
// [
//   {"firstName": "Jane", "lastName": "Orman"},
//   {"firstName": "Lily", "lastName": "Marlowe"}
// ]
// with this option with field set to "firstName" and values set to ["Jane"] should
// return
// [{"firstName": "Jane"}]
func NewQueryOptionFieldExactMatch(field string, values []string) QueryOption {
        return QueryOptionFieldExactMatch{field, matches}
}
    
type QueryOptionFieldExactMatch struct{
        field   string
        matches []string
}
    
// NewQueryOptionPage returns a QueryOption which specifies what page of
// Entites to retrieve.
func NewQueryOptionPage(limit, page uint64) QueryOption {
        return !ueryOptionPage{limit, page}
}
    
type QueryOptionPage struct{
        limit, page uint64
}
    
// NewQueryOptionPage returns a QueryOption which specifies how the results will
// be sorted.
func NewQueryOptionSort(field string) QueryOption {
        return QueryOptionSort{field}
}
    
type QueryOptionSort struct{
        field string
}

And then an example of how an implementation of Querier might handle QueryOptions:

func (m *MyQuerier) Query(ctx context.Context, opts ...eh.QueryOption) (*eh.QueryResults, error) {
        var limit uint64
        var page uint64
        var where string
            
        for _, opt := range opts {
                switch t := opt.(type) {
                case eh.QueryOptionPageLimit: // Example of built-in QueryOption.
                        limit = t.limit
                        page = t.page
                case MyWhereClause: // Example of custom QueryOption.
                        where = t.where
                default:
                        return nil, fmt.Errorf("unsupported type: %T", opt.(type))
                }
        }
            
        // Use DB-specific logic along with the variables declared at the top
        // of this function to make a query.
        
        // ...
            
        return &eh.QueryResults{entities, total}, nil
}

A few notes about this:

1. The QueryOption idea is heavily inspired by gRPC DialOptions, which are also passed as variadic arguments.
2. The interface (in query.go, perhaps) can expose some general QueryOptions itself. The implementations of Querier can choose to support them or not, and the implementations may also expose their own QueryOptions. This way, implementations of Querier may be extensible without breaking the interface API, so we can write tests for the way our transport layer (e.g., RPC or HTTP) uses the Querier using a mocked interface. As an example, we may achieve backwards compatibility (probably) with any applications using the MongoDB driver with FindCustom currently by exposing a queryOptionMongo that contains a func(*mgo.Collection) *mgo.Query, which is what FindCustom does anyway.
3. There is no method exposed for accessing a single record (i.e. Find is gone), but combining NewQueryOptionID with a check for an empty slice of entities returned from Query can achieve the same functionality. This keeps the interface very simple.
4. Paging is a first-class citizen here since as I was mentioning above, it’s really a must have for any large dataset.
5. This doesn't solve at all the problem I mentioned about aggregate queries (like SQL's group by), but potentially that could be solved by a separate Aggregator interface. That also may just need to be very custom.

This is all just stuff I’ve thought about in the last two days or so, and I am very open to any and all feedback. Everything I’m trying to do here is in service of making it easier to work within the Event Horizon framework so that when using it we’re thinking more about how to do CQRS well rather than how to do something or other with any specific database, so I’m especially interested to hear suggestions about how we can get closer to that ideal.

Also thanks to @hryx, with whom I’m working on all of these ideas. Please chime in if I've missed anything.

[screwyprof]

@benmai Really good points covered.

I've had some experience implementing some Query engines and I came up with the following interfaces:

// Query defines query parameters.
type Query interface {
	QueryID() string
}

// QueryHandler handles a query.
type QueryHandler interface {
	// Handle handles the given query and returns the report.
	Handle(ctx context.Context, q Query, report interface{}) error
}

The typical usage example may look as follows:

// query
type GetAccountShortInfo struct {
	Number string
}

func (r GetAccountShortInfo) QueryID() string {
	return "GetAccountShortInfo"
}

// report (read side model)
// Account An Account representation.
type Account struct {
	Number  string
	Balance money.Money
	Ledgers []Ledger
}

// somewhere in the application service where results are needed
func ShowAccount {
	accountReport := &report.Account{}
	err := h.queryBus.Handle(context.Background(), query.GetAccountShortInfo{Number: number}, accountReport)
       // accountReport will have all the query results
     
}

Obviously the query structure can hold all the fields needed for the request: limit, filters, offset, etc...
You may for example implement some CriteriaQuery and an SQLQueryHandler which would satisfy the interfaces above :)

You may look at a simplified examples I've made.