Data Bindings#
When deploying Microservices, you need to create a binding linking sessions to data.
Deployments can have more than one data service for different backends. A request for data needs to hit the correct data service, and use a meaningful identifier — for example a file path, or query expression. This binding is captured in the Session Service and the Gateway Service uses it to proxy incoming requests.
Info
Data Bindings are not needed for RTA Server.
Request Flow#
Typical ATLAS request flow:
- Fetch the session model —
/rta/v2/sessions/<identity> - Fetch the configs bound to the session —
/rta/v2/configs/<config_identifier> - Fetch samples and events based on the configs —
/rta/v2/sessions/<identity>/data/<type>/<channels>
For RTA Server, these activities are all handled by a single process, so no data binding is needed.
But for a microservices deployment, ATLAS talks to the Gateway Service which:
- Fetches the session model from the Session Service
- Fetches the configs from the Config Service
- Fetches the data from a data service (e.g. Toolkit Data Service, or Influx Data Service, or another data adapter):
- Gets the session data bindings from the Session Service
- Requests data for a data identity from a data source
- Streams the data back to the client — with transformations if indicated by the data binding
So each data binding must at least specify:
- The data source — which tells the Gateway Service which data service to talk to;
- The data identity — which is passed to the data service;
Info
The data source and data identity are private: the Gateway Service does not expose them via the REST API.
- data source is just a token — e.g.
influx; - data identity could be a file path, guid, query — or something else meaningful to the data service.
Creating Data Bindings#
Use the Session Service gRPC API to CreateOrUpdateSession and populate data_bindings:
This minimal binding will need a configured data service for the influx01 data source — e.g. Influx Data Service — and will use the specified data identity as a filter expression to select data for the session:
await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest
{
Identity = "example",
Updates =
{
new SessionUpdate
{
SetDataBindings = new()
{
DataBindings =
{
new DataBinding
{
Key = new()
{
Source = "influx01",
Identity = "someField='abc123'"
}
}
}
}
}
}
});
This pair of bindings captures discontinuous time ranges from the same data source and data identity:
await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest
{
Identity = "example",
Updates =
{
new SessionUpdate
{
SetDataBindings = new()
{
DataBindings =
{
new DataBinding
{
Key = new()
{
TimeRef = startTime,
Source = "influx01",
Identity = "someField='abc123'"
},
SessionStartTime = startTime,
SessionEndTime = startTime + 599_999_999L
},
new DataBinding
{
Key = new()
{
TimeRef = startTime + 900_000_000L,
Source = "influx01",
Identity = "someField='abc123'"
},
SessionStartTime = startTime + 900_000_000L,
SessionEndTime = startTime + 999_999_999L
}
}
}
}
}
});
Note
In this case, a time_ref is added to the key to make it unique.
The value of this additional key element is arbitrary: in this example it is matched to the data time range.
Important
As a result, the Gateway Service will need to inspect and potentially re-write data as it is streamed to the client to clip the data strictly to the specified bounds.
This is required because the RTA specification allows (and encourages) data services to serve data beyond the specified time-bounds so this clipping does not need to be performed on all requests, and so that data chunks have predictable cache coordinates.
Data transformation can impact data transfer throughput, so avoid using time bounds like this unless it is necessary.
This binding indicates that the session time is 300 milliseconds later than the data time:
await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest
{
Identity = "example",
Updates =
{
new SessionUpdate
{
SetDataBindings = new()
{
DataBindings =
{
new DataBinding
{
Key = new()
{
Source = "influx01",
Identity = "someField='abc123'"
},
SessionTimeOffset = 300_000_000L
}
}
}
}
}
});
As a result, the Gateway Service will need to re-write data as it is streamed to the client to add 300 milliseconds to all timestamps.
This is the main difference between this service and a conventional reverse proxy: it has some purpose-built data transformations.
Important
Data transformation can substantially impact data transfer throughput.
It is better to apply corrections upstream where possible.
This binding is a composite of data from two separate data services:
await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest
{
Identity = "example",
Updates =
{
new SessionUpdate
{
SetDataBindings = new()
{
DataBindings =
{
new DataBinding
{
Key = new()
{
Source = "influx01",
Identity = "someField='abc123'"
},
DataChannelBindings =
{
new DataChannelBinding
{
FirstDataChannel = 0,
FirstSessionChannel = 0,
LastSessionChannel = 49
}
}
},
new DataBinding
{
Key = new()
{
Source = "influx07",
Identity = "someOtherField='xxxyyyzzz'"
},
DataChannelBindings =
{
new DataChannelBinding
{
FirstDataChannel = 0,
FirstSessionChannel = 50,
LastSessionChannel = 59
}
}
}
}
}
}
}
});
This also involves a transformation: data from the second binding is channel-shifted so that it is in the range 50-59.
Data from the first binding can be proxied unchanged.
Important
Session time-range and configuration must reflect the result after transformation.
In this example, a session might have two configuration bindings — one for each underlying data source — and use a channel offset of 50 on the second binding.
This would mean that ATLAS requests channels in the range 0-59 and the requests are proxied accordingly by the Gateway Service.
Fetching Data Bindings#
The Gateway Service needs to be able to request data bindings from the Session Service:
curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions/example/data-bindings"
[
{
"key": {
"source": "influx01",
"identity": "someField='abc123'",
"timeRef": 0
},
"sessionStartTime": null,
"sessionEndTime": null,
"sessionTimeOffset": 0,
"dataChannelBindings": [
{
"firstDataChannel": 0,
"firstSessionChannel": 0,
"lastSessionChannel": 49
}
]
},
{
"key": {
"source": "influx07",
"identity": "someOtherField='xxxyyyzzz'",
"timeRef": 0
},
"sessionStartTime": null,
"sessionEndTime": null,
"sessionTimeOffset": 0,
"dataChannelBindings": [
{
"firstDataChannel": 0,
"firstSessionChannel": 50,
"lastSessionChannel": 59
}
]
}
]
This REST call is only used between these two services.
It is never called by ATLAS and should not be exposed in the public API since data bindings might contain sensitive information.