'interesting blog' 카테고리의 다른 글

OrangeCap Network  (0) 2011.08.25
Hack the Market - Algo Trading Experiences  (0) 2011.08.12
clkao  (0) 2011.07.05
Esper Basics  (0) 2011.07.05
IAmA  (0) 2011.07.04
Posted by karlsen
http://goo.gl/of5FJ

HFT was blamed for the huge drop in the stock market in May 2010, called the "flash crash." The Wall Street Journal reportedthat, "At one point, HFTs traded more than 27,000 contracts in just 14 seconds."

A nanosecond is 1 billionth of a second. One company, Fixnetix, recently announced that its  "microchip for ultra-low latency trading is now the world's fastest trading appliance for the financial markets. Customers are now seeing latencies as low as 740 nanoseconds through the stack (wire-to-wire)."

So what kind of language tools and techniques are programmers using to develop such high-performance systems? I suspected it would be something as close to the metal as possible, perhaps even assembly language for certain parts or some esoteric proprietary language. But some surprises emerged in readers' discussions of the previous article, with languages like Haskell and Smalltalk mentioned prominently.

However, it's no coincidence that Aleynikov was a C++ guy. As HFT expert Matt Davey, chief technology officer at Lab49, a financial services industry consulting firm, explained in an interview: "From a HFT platform perspective, C/C++ is the language of choice due to the latency requirements," he said. "The lower the latency, the more C/C++ is important."

Programmer/entrepreneur Jason Roberts told me the languages/tools used by HFT coders "really varies from firm to firm. Some use the Microsoft stack (.NET, C#, etc.), some use Java or C++ and others use more exotic languages like Ocaml."

Roberts has a unique perspective on the HFT industry, having worked as a founding partner for a series of small, proprietary trading companies and for a hedge fund, developing the trading technology. "I built everything from the order managers and feed parsers to the low latency messaging systems, trading UIs and tick capture databases, machine learning libraries and trading engines," Roberts said. "In most cases, I developed everything (or most everything) entirely myself."

Surprisingly, Roberts said he used C++ sparingly. "I never really used anything very esoteric and, in fact, tended to rely on tools that were fairly common and unsophisticated. For example, back between 2001 and 2003, I wrote two entire HFT trading systems (one for option market making and the other for equity pairs trading) using VB6, which even back then would've struck many as silly. The reality was that the applications worked, they were reliable and they were fast (since VB6 is compiled)."

"Moreover," he continued, "I was very productive using VB6, which enabled me to get a lot more done than if I had attempted to build everything using C++. In fact, I would only use C++ for the very rare algorithm that needed an extra boost of speed, and the one I remember best as being slow in VB6 was the square root function, of all things!"

Mike O'Hara, publisher of the High Frequency Trading Review, also listed the "C" languages as being prominent in the industry, along with Java, Matlab and Cuda. The latter, he said in an interview, is used for programming GPUs for an extra boost in performance, which seems to be an increasing trend (more on that later).

A Slashdot.org reader named "TheRaven64" said in the comments about my highest-paid-programmers article that "I periodically get contacted by recruiters for banks, because my CV mentions Haskell, and there's a massive shortage of Haskell programmers." 

The use of Haskell indicates that the new darling of the software development world, functional programming, plays a large part in the HFT industry. TheRaven64 said in his comments that recruiters who contacted him for possible employment in the financial industry "wanted people with any functional programming experience. A lot of these companies use their own in-house languages, but functional programming is popular because it's easy to verify functional programs, and because languages like Haskell facilitate rapid development. Cincom does good business selling Smalltalk to trading houses for a similar reason. Typically, a small improvement in a trading algorithm gives you an advantage for a day -- maybe less. Being able to go from idea to deployment in under an hour is something that Smalltalk and Haskell give you, and that's something that the financial industry values highly."

When one reader asked about the role of testing in such quick deployments, he was answered by yet another reader who said, "I think that was the point. Writing in one of these languages means that you can prove the correctness of the code, presumably automatically, so presumably very quickly."

Of course, there's much more to the nuts and bolts of HFT than the choice of a programming language. "From an OS perspective, the default is Linux, a custom build that has removed all the fat," Davey said. "From a back-testing perspective, there will be some kind of tick database -- Kdb or similar." But other languages come into play also, Davey said when I asked him about needed skills for HFT programming. "On the UI and data side, C# and Java are relevant, coupled with database knowledge."  

High-performance hardware and a tight marriage with software are also important. A reader on Hacker News with the handle "actionbrandon" noted in the comments that, "From the CS side, some of the more cutting-edge stuff I've heard of being worked on involves hacking graphics cards to get a speed edge and building data centers in the middle of the ocean for the same reason." 

That hardware angle was echoed by Davey: "GPU/FPGA [graphics processing units/Field-Programmable Gate Array] usage is not uncommon from a hardware perspective," he said.

That was confirmed by O'Hara, who said "Cuda is increasingly used for programming parallel processing algorithms [pattern matching] on GPUs. Impulse C is used for programming FPGAs."

But the highest-performing systems in the world are worth nothing if the underlying logic is unsound. Or, as Davey worded it, "Having low latency, but a strategy/model that generates no profit, isn't really going to help, and likewise the reverse."

Roberts agreed. Besides a thorough understanding of data structures, algorithms and concurrency issues, programmers "need to understand what it is they're building," he said. "One of the biggest hurdles for HFT firms is that the programmers, quants and traders don't understand one another, which tends to cause a lot of problems. An HFT programmer who really groks trading and some of the theory and math behind finance will be regarded as extremely valuable and can usually write their own ticket."

and : 또, 만약 그리하면, …와, 및, 그리고
Posted by karlsen

2011. 8. 11. 10:42 Market Data

TickBase

TickBase is a real-time database server for super-high-speed capture, storage and retrieval of financial market data.

Capture Real-Time Data and Gain Control

Market feeds provide traders with enormous volumes of real-time information. Yet traditional databases simply aren't fast enough to capture and store this avalanche of tick data, leaving most firms unable to fully utilize the tick data they already pay for. Leading Market Technologies has developed a solution to this problem - TickBase.

TickBase provides super-high-speed, real-time capture, storage and retrieval of tick data from a wide range of sources, architectures, external feeds and client internal sources. This server captures market data as it comes down the wire, stores that data, and then makes it available for immediate or subsequent analysis. You can use TickBase to:

  • Filter out consecutive, redundant, and outlier tick values from instruments showing huge volumes of activity but little movement.
  • Save space using Open-High-Low-Close (OHLC) format for instruments that you are only peripherally monitoring.
  • Capture important news headlines and internal messages.
  • Capture your firm's own trades.
Because TickBase achieves a more accurate and manageable database, reduces CPU load, and keeps you informed, TickBase places you in control.


TickBase Benefits

High speed

Traders and analysts demand access to the most recent data, especially during periods of high-volume trading and significant market changes. TickBase can capture very high rates of data -- enough to handle the most demanding market volumes. In contrast, commonly used SQL and other relational databases can get swamped in periods of peak market activity. Only TickBase can deliver the information you need -- now.

Faster data access

TickBase is efficient. It can be used to capture data on over 50,000 different instruments simultaneously on a dedicated mid-range UNIX server. TickBase serves up intra-day tick data to client application "on demand" -- on any instrument, without limit and without the time delays of going to external sources. A typical TickBase configuration will serve up one megabyte of tick history in one second. People use TickBase to help make crucial decisions faster.

Scalable growth 

The TickBase is parallel and scalable. You can process a virtually limitless amount of information by daisy-chaining TickBase servers together.

Data protection

TickBase is deployed on a server, not at the client application level, so the data reservoir remains safeguarded from power outages and other accidents occurring on end-users' machines.

Flexibility

LMT designed TickBase to provide fully open data access. Use pre-installed interfaces for the most popular real-time data distribution platforms or create a custom interface using the open TickBase API to link TickBase to any data source.

Easy access

Access TickBase using MS Excel, LMT's EXPO, TickView, or use TickBase's flexible API to bring data into the analytical package of your choice as well as your own applications.



TickBase Features

Disk Handling

TickBase employs a proprietary architecture and data storage strategy for speed and space efficiency. TickBase makes it possible for you to write to many disk partitions simultaneously, creating scalable databases distributed over many hard drives on a single machine.

Filters

TickBase can be easily configured to filter out bad or missing data. Among TickBase's filter types are:

  • intervals between ticks
  • context-sensitive percentage change between ticks
  • absolute value (high and/or low)
  • removal of identical values

Interfaces

LMT designed TickBase as an open client/server solution. It integrates easily with in-house proprietary systems as well as off-the-shelf software. TickBase provides pre-installed interfaces to:

  • Reuters TRIARCH SSL
  • Tibco TIB
An open, portable, C source code API allows you to create a custom interface to any data source such as:
  • Tick-by-tick market feeds
  • Internally created information
  • News wires
  • Trades and inventory
  • Client transaction

Database cleansing

The TickBase database administrator facility clean up "dirty" data upon command. For example, define an outlier value, and TickBase will remove all ticks greater than, less than, or equal to the specified value for a specified time frames.

Database merging

You can merge data from multiple databases into TickBase easily because TickBase recognizes a wide range of ASCII-type data file formats. Just define a record type and modify the API to deliver that database to TickBase for storage. If you choose, you can specify a file or list of date/time/ value/ records for merging into TickBase.

Automatic database transfers

Transfer data to or from TickBase automatically by setting TickBase to call another process at periodic intervals.

Customization 

If you wish to modify TickBase, this can be done by taking advantage of its programming interface in source code form.

Warehousing Market Data 

TickBase lets you specify the number of days after which you will archive data, on an instrument-by-instrument basis.

Standard Storage Options for Tick Data 

Using TickBase you can store market and trade data as raw ticks, bars of arbitrary period, the closing value of the day, the CHLO of the day, and/or an average value for the period. You can also conduct "one-shot" queries to acquire a tick value at a given time or list of times during the day.

Platform Availability

TickBase currently runs on the following:

  • SUN Solaris
  • HP PA-RISC 9000 Series running HP-UX

 

'Market Data' 카테고리의 다른 글

TIP. Wallstyouth, May 25, 2007  (0) 2011.08.25
Tick(Time Series) Database Discussion  (0) 2011.08.12
TREP-VA for Quantitative Trading  (0) 2011.08.11
TREP-VA for TCA  (0) 2011.08.11
TREP-VA for Compliance  (0) 2011.08.11
Posted by karlsen

블로그 이미지
Pricing, hedging, risk-managing a complex derivative product
karlsen

태그목록

공지사항

Yesterday
Today
Total

달력

 « |  » 2024.7
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31

최근에 올라온 글

최근에 달린 댓글

최근에 받은 트랙백

글 보관함