1. Apache Ignite Create Table
2. Spark Create Table
3. Ignite Create Table If Not Exists
4. Apache Ignite Create Temp Table

1 day ago  In the table insert you can see the vendors, sorted by Net Score and the Shared N column, which represents the number of shared accounts in the data set. On both dimensions, bigger is better. Start with an outline. Every presentation you create should start with an outline. Maybe you don't have tables for your caches. By default, Apache Ignite will not create tables in its caches - i.e. They are not mapped to SQL but work in key-value mode. There are basically three options to make tables in Ignite - CREATE TABLE, Query Entities, and Indexed Types + annotations.

File: get_and_put.py.

Key-value¶

Type hints usage¶

File: type_hints.py

As a rule of thumb:

• when a pyignite method or function deals with a single value or key, ithas an additional parameter, like value_hint or key_hint, which acceptsa parser/constructor class,
• nearly any structure element (inside dict or list) can be replaced witha two-tuple of (said element, type hint).

Refer the Data Types section for the full listof parser/constructor classes you can use as type hints.

Scan¶

File: scans.py.

Cache’s scan() method queries allows youto get the whole contents of the cache, element by element.

Let us put some data in cache.

scan() returns a generator, that yieldstwo-tuples of key and value. You can iterate through the generated pairsin a safe manner:

Or, alternatively, you can convert the generator to dictionary in one go:

But be cautious: if the cache contains a large set of data, the dictionarymay eat too much memory!

Do cleanup¶

Destroy created cache and close connection.

SQL¶

File: sql.py.

These examples are similar to the ones given in the Apache Ignite SQLDocumentation: Getting Started.

Setup¶

First let us establish a connection.

Then create tables. Begin with Country table, than proceed with relatedtables City and CountryLanguage.

Create indexes.

Fill tables with data.

Data samples are taken from Ignite GitHub repository.

That concludes the preparation of data. Now let us answer some questions.

What are the 10 largest cities in our data sample (population-wise)?¶

The sql() method returns a generator,that yields the resulting rows.

What are the 10 most populated cities throughout the 3 chosen countries?¶

If you set the include_field_names argument to True, thesql() method will generate a list ofcolumn names as a first yield. You can access field names with Python built-innext function.

Display all the information about a given city¶

Finally, delete the tables used in this example with the following queries:

Complex objects¶

File: binary_basics.py.

Complex object (that is often called ‘Binary object’) is an Ignite datatype, that is designed to represent a Java class. It have the followingfeatures:

• have a unique ID (type id), which is derives from a class name (type name),
• have one or more associated schemas, that describes its inner structure (theorder, names and types of its fields). Each schema have its own ID,
• have an optional version number, that is aimed towards the end usersto help them distinguish between objects of the same type, serializedwith different schemas.

Unfortunately, these distinctive features of the Complex object have few to nomeaning outside of Java language. Python class can not be defined by its name(it is not unique), ID (object ID in Python is volatile; in CPython it is justa pointer in the interpreter’s memory heap), or complex of its fields (theydo not have an associated data types, moreover, they can be added or deletedin run-time). For the pyignite user it means that for all purposesof storing native Python data it is better to use IgniteCollectionObjector MapObject data types.

However, for interoperability purposes, pyignite has a mechanism of creatingspecial Python classes to read or write Complex objects. These classes havean interface, that simulates all the features of the Complex object: type name,type ID, schema, schema ID, and version number.

Assuming that one concrete class for representing one Complex object canseverely limit the user’s data manipulation capabilities, all thefunctionality said above is implemented through the metaclass:GenericObjectMeta. This metaclass is usedautomatically when reading Complex objects.

Here you can see how GenericObjectMeta usesattrs package internally for creating nice __init__() and __repr__()methods.

You can reuse the autogenerated class for subsequent writes:

GenericObjectMeta can also be used directlyfor creating custom classes:

Note how the Person class is defined. schema is aGenericObjectMeta metaclass parameter.Another important GenericObjectMeta parameteris a type_name, but it is optional and defaults to the class name (‘Person’in our example).

Note also, that Person do not have to define its own attributes, methods andproperties (pass), although it is completely possible.

Now, when your custom Person class is created, you are ready to send datato Ignite server using its objects. The client will implicitly register yourclass as soon as the first Complex object is sent. If you intend to use yourcustom class for reading existing Complex objects’ values before all, you mustregister said class explicitly with your client:

Now, when we dealt with the basics of pyignite implementation of ComplexObjects, let us move on to more elaborate examples.

Read¶

File: read_binary.py.

Ignite SQL uses Complex objects internally to represent keys and rowsin SQL tables. Normally SQL data is accessed via queries (see SQL),so we will consider the following example solely for the demonstrationof how Binary objects (not Ignite SQL) work.

In the previous examples we have created some SQL tables.Let us do it again and examine the Ignite storage afterwards.

We can see that Ignite created a cache for each of our tables. The caches areconveniently named using ‘SQL_<schema name>_<table name>’ pattern.

Now let us examine a configuration of a cache that contains SQL datausing a settings property.

The values of value_type_name and key_type_name are names of the binarytypes. The City table’s key fields are stored using key_type_name type,and the other fields − value_type_name type.

Now when we have the cache, in which the SQL data resides, and the namesof the key and value data types, we can read the data without using SQLfunctions and verify the correctness of the result.

What we see is a tuple of key and value, extracted from the cache. Both keyand value are represent Complex objects. The dataclass names are the sameas the value_type_name and key_type_name cache settings. The objects’fields correspond to the SQL query.

Create¶

File: create_binary.py.

Now, that we aware of the internal structure of the Ignite SQL storage,we can create a table and put data in it using only key-value functions.

For example, let us create a table to register High School students:a rough equivalent of the following SQL DDL statement:

These are the necessary steps to perform the task.

1. Create table cache.

2. Define Complex object data class.

3. Insert row.

Now let us make sure that our cache really can be used with SQL functions.

Note, however, that the cache we create can not be dropped with DDL command.

It should be deleted as any other key-value cache.

Migrate¶

File: migrate_binary.py.

Suppose we have an accounting app that stores its data in key-value format.Our task would be to introduce the following changes to the original expensevoucher’s format and data:

• rename date to expense_date,
• add report_date,
• set report_date to the current date if reported is True, None if False,
• delete reported.

First get the vouchers’ cache.

If you do not store the schema of the Complex object in code, you can obtainit as a dataclass property withquery_binary_type() method.

Apache Ignite Create Table

Let us modify the schema and create a new Complex object class with an updatedschema.

Now migrate the data from the old schema to the new one.

At this moment all the fields, defined in both of our schemas, can beavailable in the resulting binary object, depending on which schema was usedwhen writing it using put() or similar methods.Ignite Binary API do not have the method to delete Complex object schema;all the schemas ever defined will stay in cluster until its shutdown.

This versioning mechanism is quite simple and robust, but it have itslimitations. The main thing is: you can not change the type of the existingfield. If you try, you will be greeted with the following message:

`org.apache.ignite.binary.BinaryObjectException:Wrongvaluehasbeenset[typeName=SomeType,fieldName=f1,fieldType=String,assignedValueType=int]`

As an alternative, you can rename the field or create a new Complex object.

Failover¶

File: failover.py.

When connection to the server is broken or timed out,Client object propagates an original exception(OSError or SocketError), but keeps its constructor’s parameters intactand tries to reconnect transparently.

When there’s no way for Client to reconnect, itraises a special ReconnectError exception.

The following example features a simple node list traversal failover mechanism.Gather 3 Ignite nodes on localhost into one cluster and run:

Then try shutting down and restarting nodes, and see what happens.

Client reconnection do not require an explicit user action, like callinga special method or resetting a parameter. Note, however, that reconnectionis lazy: it happens only if (and when) it is needed. In this example,the automatic reconnection happens, when the script checks upon the lastsaved value:

It means that instead of checking the connection status it is better forpyignite user to just try the supposed data operations and catchthe resulting exception.

connect() method accepts anyiterable, not just list. It means that you can implement any reconnectionpolicy (round-robin, nodes prioritization, pause on reconnect or gracefulbackoff) with a generator.

pyignite comes with a sampleRoundRobin generator. In the aboveexample try to replace

with

The client will try to reconnect to node 1 after node 3 is crashed, then tonode 2, et c. At least one node should be active for theRoundRobin to work properly.

SSL/TLS¶

There are some special requirements for testing SSL connectivity.

The Ignite server must be configured for securing the binary protocol port.The server configuration process can be split up into these basic steps:

1. Create a key store and a trust store using Java keytool. When creatingthe trust store, you will probably need a client X.509 certificate. Youwill also need to export the server X.509 certificate to include in theclient chain of trust.
2. Turn on the SslContextFactory for your Ignite cluster according to thisdocument: Securing Connection Between Nodes.
3. Tell Ignite to encrypt data on its thin client port, using the settings forClientConnectorConfiguration. If you only want to encrypt connection,not to validate client’s certificate, set sslClientAuth property tofalse. You’ll still have to set up the trust store on step 1 though.

Client SSL settings is summarized here:Client.

To use the SSL encryption without certificate validation just use_ssl.

To identify the client, create an SSL keypair and a certificate withopenssl command and use them in this manner:

To check the authenticity of the server, get the server certificate orcertificate chain and provide its path in the ssl_ca_certfile parameter.

You can also provide such parameters as the set of ciphers (ssl_ciphers) andthe SSL version (ssl_version), if the defaults(ssl._DEFAULT_CIPHERS and TLS 1.1) do not suit you.

Spark Create Table

Password authentication¶

To authenticate you must set authenticationEnabled property to true andenable persistance in Ignite XML configuration file, as described inAuthentication section of Ignite documentation.

Be advised that sending credentials over the open channel is greatlydiscouraged, since they can be easily intercepted. Supplying credentialsautomatically turns SSL on from the client side. It is highly recommendedto secure the connection to the Ignite server, as describedin SSL/TLS example, in order to use password authentication.

Then just supply username and password parameters toClient constructor.

Ignite Create Table If Not Exists

If you still do not wish to secure the connection is spite of the warning,then disable SSL explicitly on creating the client object:

Apache Ignite Create Temp Table

Note, that it is not possible for Ignite thin client to obtain the cluster’sauthentication settings through the binary protocol. Unexpected credentialsare simply ignored by the server. In the opposite case, the user is greetedwith the following message: