How to Programming with Kdb

A space before the last parenthesis. CSVsave:{[filename;table] filename: $[-11h = type filename;filename;`$":", filename]; @[hdel;filename;()]; h: hopen filename; (neg h) csv 0: table; hclose h; }; I'd also suggest trying a kdb IDE. Rather than having to continuously save load. e.g. qStudio ...

Installing qPython in Win7

python,kdb,q-lang

Looks like this is the culprit: g:/r/rtools/gcc-4.6.3/bin/../lib/gcc/i686-w64-mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -lpython27 And yes I would say you need to install python-dev to get the python27 libraries for linking to....

How to join splayed table in KDB?

database,kdb,q-lang

You need to map the splayed StockPrices table into memory. This can be done by using a select query: q)(`::6060)"aj[`sym`time;select from trade;quote]" / bad 'splay q)(`::6060)"aj[`sym`time;select from trade;select from quote]" / good sym time prx bid ask ------------------------------------------- aea 01:01:16.347 637.7554 866.0131 328.1476 aea 01:59:14.108 819.5301 115.053 208.1114 aea 02:42:44.724...

how to efficiiently select first or last rows from a KDB table stored on disk

.Q.ind - it takes a table and (long!) indices into the table - and returns the appropriate rows http://code.kx.com/wiki/DotQ/DotQDotind

kdb+/q: Apply iterative procedure with updated variable to a column

No need to use global vars for this - can use scan instead - see here. Example -- Generate a table - q)t:0N!([] time:5?.z.p; sym:5?`3; price:5?100f; size:5?10000) time sym price size ----------------------------------------------- 2002.04.04D18:06:07.889113280 cmj 29.07093 3994 2007.05.21D04:26:13.021438816 llm 7.347808 496 2010.10.30D10:15:14.157553088 obp 31.59526 1728 2005.11.01D21:15:54.022395584 dhc 34.10485 5486 2005.03.06D21:05:07.403334368 mho...

(kdb+/q) append to dictionary

I think it doesn't like the compound assignment groupBy,:{...} Try groupBy:groupBy,{...} The difference is that in the first case it's trying to directly alter the data in memory without creating a copy, whereas in the second case it is creating a copy of the data in memory and then re-assigning...

kdb+: replace null integer with 0

Using the fill operator (^) Example Table: q) tbl:flip`a`b!(2;0N)#10?0N 0N 0N,til 3 a b --- 0 2 1 1 1 1 1 1 Fill nulls in all columns with 0: q)0^tbl a b --- 0 2 1 1 1 1 0 1 1 0 Fill nulls only in selective columns...

How to get positions of element matches in a list in kdb?

I think you should use ? for this: q)2 5 3 1?3 5 2 1 http://code.kx.com/wiki/Reference/QuestionSymbol#find...

Unable to use launched kdb+tick demo

batch-file,cmd,kdb

That error in ticker.bat is the process attempting to execute an html file as code. Check what arguments are in ticker.bat then check that source file. I think you'll find you've download html instead of the raw text version.

How to remove element from list in KDB?

You can use except as well: q)x except 1#() a b --- 1 2 3 4 ...

How to apply max function for each row in KDB?

You can try using | q)update x|0.5 from myTable ...

Repeat last statement in the Q shell

On Linux or OS X you should use rlwrap. It is included in most Linux package repositories. On OS X I installed rlwrap via MacPorts, which requires Xcode to be installed. On Windows the q console comes with this functionality and you don't need to install anything....

KDB+ / Q table creation with foreign key

database,table,foreign-keys,q,kdb

You can use the same syntax on the column values list: q)T3:1!flip ((`id`f1 )!((1 2 3 4 5);(`T1$2 2 2 4 4))) q)T3~T2 1b Update: Again for this case we can use the same syntax on the list - q)A:enlist`T1$1 2 3 4 5 q)meta flip (`id`v1`v2)!(B,A) c | t...

timer doesn't work in qStudio

Two reasons: Show displays the value in the kdb console. There's an issue with how kdb handles remote calls involving slashes. As detailed here: http://www.timestored.com/qstudio/help/faq You should replace \t 500 with system "t 500"; and the timer will now be set. ...

How to get the total size of a table in kdb+?

-22! returns the size in bytes of in-memory objects. e.g. q)t:([] a:til 1000) q)-22!t 8031 q)/ 1000 longs = 1000*8 bytes + a small header q)t:([] a:til 2000) q)-22!t 16031 If you are interested in how memory management in kdb works I recommend this tutorial: http://www.timestored.com/kdb-guides/memory-management (Disclaimer: I wrote it.)...

In q, how to add a symbol to a symbol

In certain scenarios like this one you can use sv (http://code.kx.com/wiki/Reference/sv) q)` sv `:c:/q,a,`table `:c:/q/partofurl/table Failing that, you can always string and append q)`$":c:/q/",string[a],"/table" `:c:/q/partofurl/table ...

How to select first record prior/after a given timestamp in KDB?

database,kdb,q-lang

Select last record before the given timestamp: q)select from Price where timestamp<2014.04.14T09:30,stock=`GOOG,i=last i Select first record after the given timestamp: q)select from Price where timestamp>2014.04.14T09:30,stock=`GOOG,i=first i ...

KdB+ - HTML5 websocket deserializer not working for Long data type

html5,websocket,kdb

As rightly pointed out in the comment by user2393012, the serializer for KdB+ maps 0N(null) to the min value of the data type. Hence a null value of Long type gets serialized to -9223372036854775808. The deserializer is handling this integer min correctly for 16 bit and 32 bit values as...

kdb: inter-process bulk insert

use this: q)h (insert; `TEST;DATA) Syntax for calling function on a service from other service using handle: h(function;func_parameters) ...

Kdb+/q: Looping over initialised list

What you're doing may work but it's far from best practice. Loops and indices are not the way to go. What you're looking for is essentially test:([] name:`symbol$(); balance:`int$()); insert[`test;(`John;1001)]; insert[`test;(`Jane;2002)]; q)select sum f[balance] from test balance ------- 30.03 ...

Select value from mixed-type row in KDB

I almost don't want to give a solution here because mixing strings with integers in a single column is a terrible idea. Completely destroys performance and prevents any of the benefits that kdb offers. First and foremost, re-think your setup. If you insist on keeping it as is, you can...

KDB+ / Q Table creation with dynamic number of columns

list,table,q,kdb

A table is a dictionary from a dictionary of symbol keys to a list of equal length vectors. Dynamically create the column names using "string til". q){ `id xkey update id:i from flip (`$"vals",/:string til y)!(y#x)?\:`8 }[3;4] id| vals0 vals1 vals2 vals3 --| ----------------------------------- 0 | lkamnmmm nfnbfmkm kiblpojl onhlghno...

How do I use the k() function to receive data in kdb/q

java,c#,asynchronous,kdb

The k() function will block until it receives data from the remote socket. So if your function looked like this: compute_long_running_function:{[] r:til 1000; neg[.z.w] r } the function result will be sent over the loopback connection and be retrieved by conn.k() The alternative is you just do: results=conn.k("compute_long_running_function[]"); which will...

Inconsistent behaviour with each and peach in KDB

That was not an inconsistent behavior of peach and each. First, Functions in kdb has implicit parameters as x,y,z if not specified any. So f:{x+y} is equivalent to f:{[x;y] x+y} But f:{[a;b] a+b} will not have x,y,z as implicit parameter For more details, see section Implicit Parameters in http://code.kx.com/wiki/JB:QforMortals/functions#Implicit_Parameters Peach...

Construct q command to get metadata for all tables

database-connection,port,32bit-64bit,kdb

q)trade:([] sym: 10?`4; time:10?.z.t; prx:10?100f; sz:10?10000); q)quote:([] sym: 10?`4; time:10?.z.t; bPrx:10?100f; aPrx:10?100f; bSz:10?10000; aSz:10?10000); q)testTable:update `s#a from ([] a:til 10; b: 10?`3; c:10?.z.p); q)raze {update table:x from 0!meta x}'[tables[]] c t f a table -------------------- sym s quote time t quote bPrx f quote aPrx f quote bSz j quote...

kdb+ 32-bit, what are the consequences of the 4GB restriction?

Correct, a kdb+ instance is a whole process in its own right, with a port number set at command line by yourself. You can't do peach over a number of handles: handles:hopen each someList; {x"do some work"} peach handles You'll get noupdate, so would seem you can't do this manually....

KDB Excluding rows that have nulls

null,kdb

Either use simple list functions for easy to read code or functional form for speed. The functional form will be faster as it doesn't scan all the entire columns just those that pass each stage of the filter. q)t:flip `a`b`c`d`e!flip {5?(x;0N)} each til 10 q)t a b c d e...

KDB: user-defined aggregation function on tables

aggregate,kdb

You can use : q)select enlist MySum x2 from data Why 'select Mysum x2 from data' doesn't work? From KDB WIKI: "The following functions receive special treatment within select count,first,last,sum,prd,min,max,med,avg,wsum,wavg,var,dev,cov,cor " For Details, read: http://code.kx.com/wiki/Reference/select Section: Special functions within select ...

Using flip inside each not working in KDB

Remember that table is a list of dictionary. q)show each ([]a:1 2 3;b:4 5 6); a| 1 b| 4 a| 2 b| 5 a| 3 b| 6 http://code.kx.com/wiki/JB:QforMortals2/tables#Overview The values of each dictionary are atoms, not lists. So you can't flip a dictionary into a table that way, you have...

How to publish to KDB Ticker Plant from Java effectively

java,kdb,ticker,q-lang,exxeleron-q

There's not enough information in this message to give a fully qualified response, but having done the same with Java+KDB it really comes down to eliminating the possibilities. This is common sense, really, nothing super technical. make sure you're inserting asynchronously Verify it's exxeleron q java that is causing the...

kdb+: use string as variable name

You could use "set" but it will create a global: q){(`$"test") set 1;test}[] 1 q)test 1 or (as noted by user2393012 in the comments): @[`.;`test;:;1] If you want to avoid globals you could use some sort of namespace/dictionary/mapping: q){d:()!();d[`$"test"]:1;d`test}[] 1 ...

How can I start kdb instance in Windows?

q trade.q -p 5001 This video tutorial demonstrates downloading and installing kdb on windows: http://www.timestored.com/kdb-guides/getting-started-kdb...

KDB Query with upper in where clause

Try "upper[cola]=`ABC". Round brackets are not used for function arguments, they are used for precedence etc. Square brackets are used in function arguments. The reason round brackets worked in the second case is because the parser simply ignored them. Would be the same as "select upper cola from table" In...

Pivot table in kdb+/q

This is a self-contained version that's easier to use: tt:1000#0!trades_agg piv:{[t;k;p;v] / controls new columns names f:{[v;P]`${raze " " sv x} each string raze P[;0],'/:v,/:\:P[;1]}; v:(),v; k:(),k; p:(),p; / make sure args are lists G:group flip k!(t:.Q.v t)k; F:group flip p!t p; key[G]!flip(C:f[v]P:flip value flip key F)!raze {[i;j;k;x;y] a:count[x]#x 0N;...

Use a function in the groupby clause of a functional select

1. Yes you can, just function should be allowed/compatible to group by clause. 'xbar' is allowed. Syntax is: (function;param1;parma2;...param n) Here is one ex. q)t:([]id:til 20;v:til 20) q)select by 5 xbar id from t q) // functional form q)?[`t;();(enlist `id)!enlist (xbar;5;`id);()] 2. Trick to get functional form of simple query...

Dictionary assignment in defined namespace failing

From code.kx: A context is actually a sorted dictionary whose domain is a list of symbols with the names of the entities defined in the context If you change your code to the following, it will work as expected: q).foo:(`symbol$())!() q).foo.bar:`a`b`c!1 2 3 q).foo.bar a| 1 b| 2 c| 3...

kdb+ conditional insert: only insert when column value doesn't exist

insert-update,kdb,dotq

First thing is to have proper attributes (sort,group) on the target column which will make function faster. Now there are 2 scenarios I can think of: a) Table is keyed and target column is keyed column : In this case normal insert will work in way like your conditional insert....

How to append a single element to a list keyed in a dict

You can do dict[`b],:8 but it will automatically overwrite the dictionary. A more robust way would be @[dict;`b;,;8] or @[`dict;`b;,;8] where the latter automatically overwrites the dictionary, while the former creates a modified copy of the dictionary (in case you don't want it to overwrite the original yet)...

How to kill KDB queries in RDB or HDB?

You can set client query time out in your service: param: '-T ' reference: http://code.kx.com/wiki/JB:QforMortals2/commands_and_system_variables#Timeout_.28-T.29 From wiki: The timeout parameter (note upper case) is an int that specifies the number of milliseconds any call from a client will execute before it is timed out and terminated. The default value is...

Aggregate rdb and hdb in kdb

Say this function lives on the an independant q instance and you want all columns, for a day d + today for sym s... notice the column rearrangement. mergeFun:{[d;s] rdb:hopen`::5011; hdb:hopen`::5012; t:hdb({[d;s] `time`sym xcols delete date from select from table where date=d, sym=s};d;s),rdb({[s] select from table where sym=s};s); hclose each...

kdb+ historical database partitioning design - daily data on thousands of symbols

database-design,kdb

As @GilbertLeBlanc says, 50m is certainly not much and if that was the expected size it would simply be best (in this use-case) to splay the table (i.e. no partioning at all) along with using attributes on the columns that you will use for filtering (p# org# on ticker since...

kdb 2-way asof join (merge bid/ask tables)

You don't need aj to achieve what you want. uj combined with fills will do the job: b: 1!`time`sym`bs`bprc xcol bid / convert bid to a keyed table a: 1!`time`sym`as`aprc xcol ask / convert ask to a keyed table ba: `time xasc b uj a / join them and sort...

Updating point-in-time (bitemporal) table

You can use 'upsert' in following way: reference: http://code.kx.com/wiki/Reference/upsert Step 1. I am adding status column in 'new' table (otherwise it will be null for new rows in final table) q) new:update stamp:`timestamp$.z.z from new Step 2. Make those columns of 'new' as primary keys whose modification is criteria for...

kdb+/q: Prevent initialised tables from returning a comma

Array access with [0] or first is the correct way if you want an "atomic" variable. myvar:first exec name from test; ...

Passing Date Vector from R to KDB

I usually generate a list of dates and convert them into KDB readable by using sub function dates <- seq(as.Date("2014/11/01"), as.Date("2014/12/01"), by = "day") dates <- sub('-',".",dates);dates <- sub('-',".",dates); dates [1] "2014.11.01" "2014.11.02" "2014.11.03" "2014.11.04" "2014.11.05" "2014.11.06" "2014.11.07" "2014.11.08" "2014.11.09" "2014.11.10" [11] "2014.11.11" "2014.11.12" "2014.11.13" "2014.11.14" "2014.11.15" "2014.11.16" "2014.11.17" "2014.11.18"...

KDB / Q Resetting random elements on newly created list (numerical vs symbol/char)

list,random,q,kdb,nul

To make it generic would be a matter of using the booleans to index rather than using them to multiply (as you currently are). However, for non numerical lists you'd need to reset values to nulls rather than to zero. These nulls would have to be the correct null corresponding...

Why does KDB Ticker Plant need a log file?

It is just to restore the data mainly in case of RDB process crash which ensures system stability. You can assume it as a backing store or as a permanent storage. First let's understand the tickerplant architecture: Tickerplant takes data from a source (feed handler) , logs it in file...

How to push tables through socket

Let's understand the simple process of doing this: We have one server 'S' and one client 'C'. When 'C' calls .u.sub function, that function code connects to 'S' using its host and port and call a specific function on 'S' (lets say 'request') with subscription parameters. On getting this request,...

KDB+ / Q Accessing root namespace from namespace

namespaces,q,kdb

Context (namespace) is just a dictionary so you can use dictionary syntax. q) \d .seed q.seed) `.[`n] Reference: http://code.kx.com/wiki/JB:QforMortals/workspace_organization Check section: A Context is a Dictionary ...

type char to type num mapping

q,kdb

You can use .Q.t. q).Q.t abs type `test "s" q).Q.t abs type 5i "i" Edit: Or even better just use .Q.ty This seems to return upper case for atoms and lower case for lists. q).Q.ty `test "S" q).Q.ty `test`test2 "s" ...

KDB+ / Q Better webinterface

q,kdb,web-interface

Have you tried doth package by simon? This is probably not even close to what you describe, but it could be a starting point to implement something custom.

How to work with date and time in KDB

to remove the date, you can use the cast operator, $. To reference only the time, you can prefix $ with `time as shown below. q).z.z 2015.02.23T14:10:33.523 q)`time$.z.z 14:10:30.731 q)t:([]ts:10#.z.N;ti:.z.t-til 10) q)exec `time$ts-ti from t 00:00:00.000 00:00:00.001 00:00:00.002 00:00:00.003 00:00:00.004 00:00:00.005.. You can see more examples here. http://code.kx.com/wiki/Reference/DollarSign...

how to create bulk object with char arrays for KDB+

c++,kdb,k

A string column in kdb is considered as a general type (list of list of char). q)type each flip ([]a:string`aa`bb;b:"ab") a| 0 <-- string column b| 10 <-- char column So your kdb table definition should've been // table_def:([] name: ()) Your previous definition is expecting the column values to...

Cannot allocate memory for a column of compound floats on a partitioned table

I think this is all just a combination of the free-32bit memory limitation, the fact that your row counts are possibly large and the fact that (unavoidably) something must be pulled entirely into memory when retrieving data from a column, whether it is the column itself that gets entirely pulled...

Use value to reference a variable in a function

value will always work on global scope. If you really need this, maybe make use of workspace variables, e.g. .a.b:1 ... I don't have a q instance to hand to test if that works but i'm almost sure it does....

Quicksort in Q/KDB+

Lets do it step by step . I assume you have basic understanding of Quick Sort algo. Also, there is one correction in code you mentioned which I have corrected in step 5. Example list: q)x: 1 0 5 4 3 Take a random element from list which will act...

No such host is known using AquaQ

Terry you were right actually. It turns out q requires this "/" in paths not "\". I had "\" in the path for the environment variables. Johnny Press answered this for me here

kdb+: group by and sum over multiple columns

How about this? get select first time, avg colA, sum colB, first colC by sums colC<>prev colC from table ...

How to convert character array into list in kdb

You are missing 'each' here. Use: select enlist each column_with_character_arr from table ex: q) t:([]v:("abc";"xyz")) q) select enlist each v from t Why length error on select (),v from t? Because comma(,) in select query separates different columns we want in output, i.e. select col1,col2 from tbl So in query...

KDB: apply dyadic function across two lists

function,kdb

You have asked for the cross product of your argument lists, so the correct answer is raze F ./: xList cross yList ...

Saving a kdb+ “unmappable” table to disk

kdb,q-lang,rdb

You can manually save these tables using the following steps: q)t:([] a:1 2 3; f:(1 2.0;(3 4.;5 6.); 7. 8.)) q)t a f --------- 1 1 2 2 3 4 5 6 3 7 8 q)`:t/ set t k){$[@x;.[x;();:;y];-19!((,y),x)]} 'type q.q)) q.q))\ q) Fails as nested type. q)`:t/ set select...

General position find function in q

matrix,position,find,kdb,q-lang

How's this look? I think it should work for all two-dimensional lists, ragged or rectangular, and also for vectors. (I haven't worked out a version for arbitrary dimensions yet.) q)position:{{$[type x;enlist each where x;raze flip each flip(til count x;raze each .z.s each x)]}x=y} q)t 1 -1 1 -1 3 4...

KDB+ / Q unique random values with variable for amount

random,q,kdb

This will give you num unique numbers between 0 and 9. q)(neg num)?10 ...

q: `': The pipe is being closed`

@[loopSizes; 0; {0N!"error thrown, returning zero";}] Not sure why you were setting z to 0 in the 2nd argument of @ - just 0 suffices and gets passed to loopSizes. (so long as loopSizes is a monadic function). 0N! prints to console. Also see -1! to print to standard out....

KDB projectiopn multiple variable

Here is my understanding of your doubt: You have a list of inputs as L:(inputs1;inputs2;...) , where inputs1 is a list of (date;symbol;starttime;endtime) and you want to apply 'test' function on each input list in 'L'. For this, KDB provide 'dot' operator. Ex: q) f:{[a;b;c] a+b+c} q) f . (1...

kdb - qStudio multiple line execution

For running multi-line in QStudio, use semicolon for line ending. For ex, following 2 lines will not run together in qstudio using Ctrl+E: a:1 b:2 `type error. If you look in history tab, it sends command (a:1b:2) So use a:1; b:2; Now '\l' doesn't work with semicolon ending, it gives...

Difference when you query from KDB HDB and KDB RDB

Off the top of my head: Partitioned HDB tables will have a "virtual" date column RDB tables (generally) won't have a `date column The virtual "i" column behaves differently for Partitioed HDB tables (http://code.kx.com/wiki/DotQ/DotQDotind) HDB tables (unless stored flat/serialised) are not immediately pulled entirely into memory, data is read on...

checking in list from kdb instance

Something like the following will work: ans:hdl({select count i by date,sym from trade where date=xxx, sym in x};res) @mollmerx is correct about having res defined locally and not on the remote process, although my query above is preferable to a functional select. ...

How to use .year inside function in KDB?

As per this page on code.kx, this behavior is a quirk of q. To get around this, you can use the cast function. q)f:{[x] :`year$x} q)f[myDate] 2014i ...

How to join multiple tables