Extracting Protocol Format as State Machine via Controlled Static Loop Analysis. (arXiv:2305.13483v1 [cs.CR])

Reverse engineering of protocol message formats is critical for many security
applications. Mainstream techniques use dynamic analysis and inherit its
low-coverage problem -- the inferred message formats only reflect the features
of their inputs. To achieve high coverage, we choose to use static analysis to
infer message formats from the implementation of protocol parsers. In this
work, we focus on a class of extremely challenging protocols whose formats are
described via constraint-enhanced regular expressions and parsed using
finite-state machines. Such …

analysis applications critical dynamic dynamic analysis engineering features high inputs loop low machine message problem protocol reverse reverse engineering security state static analysis techniques