This page will contain a detailed description - for developers - of how the Binary Protocol will be laid out.
With the initial discussion mcell and hackish discussed the ideas of how frames should be defined.
TODO: search for opendiag and see if there's something (standard) we could use (they apparently don't have a wiki, so it'll suck a bit to catch up); someone could subscribe and keep an eye?
Aims
- robustness (reliability): transmission errors handled gracefully
- compactness: reasonable data-footprint in flash and via network
- reasonable CPU usage (no compression, avoiding byte stuffing, reasonable checksum, no ECC)
- flexibility: supports several devices on the same bus
- preferrably same code used for
- logging into flash (the FLASH applies ECC internally, transparent to the user) as
- logging to network eg. to PC or
- runtime sensor-data to another board
- maybe even for dumping configuration.
- preferrably industrial standard
SerialComm/SIPR is already implemented (by Hackish). SIPR is much better than MegaTune protocol, since it supports reliability. Unfortunately SIPR does not support flexibility (multiple devices on a bus, or complex topology).
Real CAN supports flexibility, but CAN does not support compactness (8byte load is a bit small for logging, for which the 6 byte overhead is a bit prohibitive) and real CAN controller not available on many controllers.
Solution
In order to make an error resilient system we have decided on the following.
- we pack CAN-like payloads N * (4byte ID + 8 byte value) into CRC protected frames
- we define gigapacks, mainly for logging, with length = N * 12byte. This allows efficient storage of runtime-data logs
CAN-IDs
CAN-ID is a 29bit address that identifies the data. The beauty of CAN is that the usage is not defined. CAN-ID can identify the source of the data, or the destination. It can be a command, which has some parameters (max 8 bytes of payload).
We propose to use
- 13bits (most significant part of the ID) for data-type
- and 16 bits (least significant part of the ID) for device address. Device addresses are unique on a network.
When packing 29bit CAN-IDs into UART frames (this part consumes 4 bytes = 32 bits), we use the remaining 3 bits to mark gigapacks:
- 0: normal CAN packet (4+8 byte)
- eg. the configuration values and requests, that are now arranged in pages of 256 bytes each, will be 32 different CAN-IDs, 8 bytes each
- while format of configwrite and configreport command is trivial, we want to make configrequest possible without sacrificing 1 bit from the precious 29 bits: we steal 256 combinations from the 65536 possible (masking the upper byte of the 16 bit to 0x00) hardware addressed, and these mean config-request. It is possible that more than one devices will reply, but that's harmless.
- 1..5: gigapack, length: 1..5 x 24 byte (including this 4 byte ID)
- the runtime variable log would probably be 72 or 96 byte long gigapack
- 6..7: reserved (for longer packets)
- All data transferred within frames (length: header + Nx12 bytes + 2byte CRC).
- The frame contains as compact info as possible:
- marker byte (eg. \n)
- N = payload length (in 12 byte units)
- production data. N*12 bytes, see above.
- 2 CRC16 bytes as defined in the avrlibc CRC_update() routine. Note that this is independent of the CRC7 that is used when talking to MMC in SPI mode.
Arbitration
Some devices are slave-only. A slave can only talk if it was requested by the master. However, than it is permitted to send it's queued messages, including messages to other slaves.
Some devices (eg. ARM and PC) can act as master. Only one master at a time is active. If a master-capable device cannot hear any other master for master_existence_timeout (appr. 20..100msec), takes over the bus (involving some randomness to decrease chance of collision), and grants bandwidth to each slave.
example
- ARM is master
- PC is slave
- v3.x is slave
PC wants to save a config value to v3.x.
- PC must wait for it's turn, when ARM requests PC to speak.
- PC sends the save-config request to v3.x
- v3.x saves the config, and generates the configreport reply. v3.x cannot send the reply immediately, as it's not his turn
- in the meantime, maybe ARM asks several other devices to speak
- after ARM requests v3.x to speak, v3.x can send his configreport which the PC hears too.
Secondary problems
- Extended address: besides the 16 bit HW address, there is an extended full HW-address that is unique worldwide. The lower 16 bits is the bootloader address.
- device discovery involves messages that only apply to devices that match certain address/mask. Originally we require the installer to record (in explicite configuration) every HW (including their addresses!) added to the segment. Later we might provide full autodetection and plug and play utilizing smart master that can detect collisions (CRC error).
- the flash might have a defined 16 bit address to avoid collision: normally the 16bit bootloader address is inherited
Bootloader HW-indication changes
Shaking the TX in hope the TX is loopbacked to RX is not desirable. The bootloader will wait (listening to the bus) bootloader_inactive_timeout (appr 200msec) than start the application, unless:
- of course, if the app entered the bootloader intentionally, the bootloader will stay in bootloader mode
- special stay_in_bootloader frame makes it stay in bootloader mode
- special start_app frame makes it start app immediately
Above special commands are address/mask addressable selectively.
Gigapacks
TODO: clean this
- several common frame_type_description-s must live in flash. Can be listed and all can be queried for details.
- at least one (maybe more later) frame_type_description can be defined from menu. This has the same structure as the one in flash, but lives in SRAM (which is more expensive).
- the frame_type_description refers to variables, that are boardtype+application specific. Note: 256 variables is not enough for a given application, so 2 bytes is likely. Also helpful if arrays should be needed. Should this be flat or hierarchical?
A frame is defined as
| frame_marker | frame_lenght | data | CRC16 |
| 0x0A | N | N * 12 byte DATA | CRC |
A frame_type_description payload is defined as
- length (special value might mean it is computed ?)
- description text
and a repetition of:
- data-type (=> determines data-length)
The frame_type_description is packaged in a frame (using a special frame_type) when transmitted or stored (tricky?).
Byte stuffing or not?
Byte stuffing (escaping the frame_marker byte) is not needed.
- only valid frames are processed in any case (we cannot avoid the CRC anyway - except maybe in flash)
- we save bytes without byte stuffing (especially nice in MMC flash - GenBoard/LoggerIntegration )
- we save computational power without byte stuffing in the normal (optimistic) case (when no communication or storage errors)
- in the worst case (if some bytes are lost or corrupted) it might take slightly more computation to find the valid frames without byte-stuffing: if the payload also contains the frame_marker, some extra frame-candidates will be examined (checksum calculation) and dropped if it was found to be a fake frame_marker. Processing continues onto next candidate. Note that even with byte stuffing, it is unavoidable that sometime fake frame candidates are examined because of fake frame_marker-s occuring out of the blue (corruption).
It seems we get better overall performance without byte stuffing.
Acknowledge
We better support
- Sending acknowledge
- Requesting acknowledge - perhaps this is redundant?
- Protocol could define frame_types which require acknowledgement. All non-returning (eg "get" messages) messages should be acknowledged.
Some sequence number is required for this.
We must support the acknowledge piggybacked in a useful frame (that might possibly be empty) to avoid extreme acknowledge overhead. Overhead is one cost of robustness and security. It'll always be a trade off - we need not even use a frame... a single bit or marker byte is plenty.
Lets look at 2 usecases for updating a value from tuning software. Steps 1 & 2 are about all that's done right now, but there's no verification that anythng worked (correct me if I'm wrong!). That's not good enough.
Usecase without ack:
- Software sends command to set VE table [1][2] to 35.
- ECU receives command, and updates VE table [1][2].
- Software sends command to read VE table [1][2].
- ECU sends back VE table [1][2].
- Software tests that command worked.
Usecase with ack:
- Software sends command to set VE table [1][2] to 35.
- ECU receives command, updates VE table [1][2], and sends ack.
- Software receives ack and knows command worked.
Are we trying too hard to reinvent a reliable message passing protocol? If we're talking about using CRC functions and ack and so on anyway... CAN already works, and can be implemented over a 1-wire interface (see SAE J2411). And we need it for next gen hardware. We should get it figured out on some level now.
Some bytes are lost forever - this condition must be handled gracefully in any case.
We expect there to be a defined timeout where the receiver will give up and request that a sender re-send the frame (issuing the same - idempotent - command again). Note that for the PC-GenBoard communication the PC will take care of this. The GenBoard doesn't care if the PC does not get the reply (the PC will request again).
Please list any operations (firmware commands) that are not idempotent (issuing again can have side-effects).
- mcb.. (going into bootloader mode: repeating unnecessarily can start stupid action in bootloader - chance is low )
- mcB.. change baudrate but remain in ECM application
- ...
---
See also:
- GenBoard/LoggerIntegration/DataFormat (TODO: cleanup, this page is newer)
- GenBoard/LoggerIntegration
- [MODBUS standard]