cmdline: fix dynamic tokens interface

cmdline: fix dynamic tokens interface

Support for dynamic tokens was added in order to implement the flow command
in testpmd, for which static tokens were not versatile enough due to the
large number of possible parameter combinations.

However, due to its reliance on a temporary array to store dynamic tokens,
this interface suffers from various limitations that need to be addressed
in order to implement more commands in the future:

- The maximum number of dynamic tokens is determined at compilation time
  (CMDLINE_PARSE_DYNAMIC_TOKENS). The larger this value, the more stack
  space is wasted (one pointer per potential token, i.e. 1kB of stack space
  on 64-bit architectures with the default value).

- This temporary array is actually a cache in which entries already present
  are not regenerated. This behavior is not documented, which makes dynamic
  tokens practically unusable by applications as they do not know which
  token is current.

- The cache does not really reduce the number of function calls needed to
  retrieve tokens, it was mainly deemed useful to provide context about
  other tokens to the generator callback.

- Like testpmd, most users will likely use repeated pointers to a fixed
  token header structure (cmdline_token_hdr_t), with internal context-aware
  callbacks that do not need to look at other entries; knowing the index of
  the current token is enough.

Getting rid of the temporary array and properly documenting usage of the
token generator callback greatly simplifies this interface.

Fixes: 4fffc05a2b2c ("cmdline: support dynamic tokens")
Fixes: 19c90af6285c ("app/testpmd: add flow command")
Cc: stable@dpdk.org
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>