Sunday, 28 December 2014

Indonesia halts search for missing AirAsia plane as night falls

 Indonesia called off until first light a search for an AirAsia plane with 162 people on board that went missing on Sunday after pilots asked to change course to avoid bad weather during a flight from Indonesia's Surabaya city to Singapore.

Indonesia AirAsia Flight QZ8501, an Airbus 320-200 carrying 155 passengers and seven crew, lost contact with Jakarta air traffic control at 6:17 a.m. (2317 GMT on Saturday). No distress signal had been sent, said Joko Muryo Atmodjo, an Indonesian transport ministry official.

On board were 155 Indonesians, three South Koreans and one each from Singapore, Malaysia and Britain, plus a French pilot, the airline said in a statement, correcting earlier information.

Tatang Kurniadi, head of Indonesia's National Committee of Safety Transportation, expressed hope of locating the aircraft quickly and said it was too early to detect any of the so-called electronic pings from its black box recorder.

"We are using our capacity to search on sea and land. Hopefully we can find the location of the plane as soon as possible," he told a news conference.

"What I need to emphasize is until now, we have not found out how the plane fell or what kind of emergency it was."

Indonesia AirAsia is 49 percent owned by Malaysia-based budget carrier AirAsia (AIRA.KL), which has had a clean safety record since it began operating 13 years ago. The AirAsia group also has affiliates in Thailand, the Philippines and India. The aircraft had accumulated approximately 23,000 flight hours in some 13,600 flights, according to Airbus (AIR.PA).

The pilots of QZ8501 "was requesting deviation due to en-route weather before communication with the aircraft was lost," the airline said in a statement. Singapore, Malaysia, Britain, South Korea and Australia offered to help in the search and any investigation. Malaysia said it was sending vessels and a C130 aircraft while Singapore had also sent a C130. Australian Prime Minister Tony Abbott said a P3 Orion aircraft was on standby if needed.


Flight QZ8501 was between Tanjung Pandan on Indonesia's Belitung island and Pontianak, in West Kalimantan province on Borneo - almost halfway between Surabaya and Singapore - when it went missing. There was bad weather over Belitung at the time and the aircraft had been flying at 32,000 feet before asking to fly at 38,000 feet to avoid clouds.

In both Surabaya and Singapore, anxious relatives of people on the plane awaited news.

Louise Sidharta was at Singapore's Changi Airport waiting for her fiancée to return from a family holiday.

"It was supposed to be their last vacation before we got married," she said.

A man named Purnomo told TVOne in Surabaya of a lucky escape.

"I should have been on the flight," he said. "We, seven people, had planned to go to Singapore for vacation but this morning I had an emergency. I had my passport in hand."

Tony Fernandes, chief of Malaysia's AirAsia, said he was heading to Surabaya.
 AirAsia plane
"My only thoughts are with the passengers and my crew. We put our hope in the SAR (search and rescue) operation and thank the Indonesia, Singapore and Malaysian governments," he said on Twitter.

AirAsia swapped its distinctive bright red logo for a grey background on its website and social media accounts.

The incident comes during a troubled year for Malaysia-affiliated airlines. Malaysian Airlines Flight MH370 went missing on March 8 on a trip from Kuala Lumpur to Beijing with 239 passengers and crew on board and has not been found.

On July 17, Flight MH17 was shot down over Ukraine, killing all 298 people on board.

Indonesia AirAsia has a fleet of 30 Airbus A320s. The missing plane has been in service for just over six years, according to

All AirAsia-branded airlines operate aircraft made by Airbus, which has orders for several hundred planes from the group. AirAsia is considered one of the European planemaker's most important customers.

Friday, 26 September 2014

Pig Operators - Hadoop

Basic Operators

Operator Description Example
Arithmetic Operators +, -, *, /, %, ?: X = FOREACH A GENERATE f1, f2, f1%f2;
X = FOREACH A GENERATE f2, (f2==1?1:COUNT(B));
Boolean Operators and, or, not X = FILTER A BY (f1==8) OR (NOT (f2+f3 > f1));
Cast Operators Casting from one datatype to another B = FOREACH A GENERATE (int)$0 + 1;
B = FOREACH A GENERATE $0 + 1, $1 + 1.0
Comparison Operators ==, !=, >, <, >=, <=, matches X = FILTER A BY (f1 == 8);
X = FILTER A BY (f2 == ‘apache’);
X = FILTER A BY (f1 matches ‘.*apache.*’);
Construction Operators Used to construct tuple (), bag {} and map [] B = foreach A generate (name, age);
B = foreach A generate {(name, age)}, {name, age};
B = foreach A generate [name, gpa];
Dereference Operators dereference tuples ( or tuple.(id,…)), bags ( or bag.(id,…)) and maps (map#’key’) X = FOREACH A GENERATE f2.t1,f2.t3 (dereferencing is used to retrieve two fields from tuple f2)
Disambiguate Operator ( :: ) used to identify field names after JOIN, COGROUP, CROSS, or FLATTEN operators A = load ‘data1′ as (x, y);
B = load ‘data2′ as (x, y, z);
C = join A by x, B by x;
D = foreach C generate A::y;
Flatten Operator Flatten un-nests tuples as well as bags consider a relation that has a tuple of the form (a, (b, c)). The expression GENERATE $0, flatten($1), will cause that tuple to become (a, b, c).
Null Operator is null, is not null X = FILTER A BY f1 is not null;
Sign Operators + -> has no effect, – -> changes the sign of a positive/negative number A = LOAD ‘data’ as (x, y, z);

Relational Operators

Operator Description Example
COGROUP/GROUP Groups the data in one or more relations. The COGROUP operator groups together tuples that have the same group key (key field) A = load ‘student’ AS (name:chararray,age:int,gpa:float);
B = GROUP A BY age;
CROSS Computes the cross product of two or more relations X = CROSS A,B A = (1, 2, 3) B = (2, 4)
DUMP X; (4, 2, 1) (8, 9)
(1,2,3,2,4) (1, 3)
DEFINE Assigns an alias to a UDF or streaming command. DEFINE CMD `perl – nameMap` input(stdin using PigStreaming(‘,’)) output(stdout using PigStreaming(‘,’));
A = LOAD ‘file’;
DISTINCT Removes duplicate tuples in a relation. X = DISTINCT A; A = (8,3,4)
DUMP X; (1,2,3)
(1,2,3) (4,3,3)
(4,3,3) (4,3,3)
(8,3,4) (1,2,3)
FILTER Selects tuples from a relation based on some condition. X = FILTER A BY f3 == 3; A = (1,2,3)
DUMP X; (4,5,6)
(1,2,3) (7,8,9)
(4,3,3) (4,3,3)
(8,4,3) (8,4,3)
FOREACH Generates transformation of data for each row as specified X = FOREACH A GENERATE a1, a2; A = (1,2,3)
DUMP X; (4,2,5)
(1,2) (8,3,6)
IMPORT Import macros defined in a separate file. /* myscript.pig */
IMPORT ‘my_macro.pig’;
JOIN Performs an inner join of two or more relations based on common field values. X = JOIN A BY a1, B BY b1;
(1,2,1,3) A = (1,2) B = (1,3)
(1,2,1,2) (4,5) (1,2)
(4,5,4,7) (4,7)
LOAD Loads data from the file system. A = LOAD ‘myfile.txt’;
LOAD ‘myfile.txt’ AS (f1:int, f2:int, f3:int);
MAPREDUCE Executes native MapReduce jobs inside a Pig script. A = LOAD ‘WordcountInput.txt’;
B = MAPREDUCE ‘wordcount.jar’ STORE A INTO ‘inputDir’ LOAD ‘outputDir’
AS (word:chararray, count: int) `org.myorg.WordCount inputDir outputDir`;
ORDERBY Sorts a relation based on one or more fields. A = LOAD ‘mydata’ AS (x: int, y: map[]);
SAMPLE Partitions a relation into two or more relations, selects a random data sample with the stated sample size. Relation X will contain 1% of the data in relation A.
A = LOAD ‘data’ AS (f1:int,f2:int,f3:int);
X = SAMPLE A 0.01;
SPLIT Partitions a relation into two or more relations based on some expression. SPLIT input_var INTO output_var IF (field1 is not null), ignored_var IF (field1 is null);
STORE Stores or saves results to the file system. STORE A INTO ‘myoutput’ USING PigStorage (‘*’);
STREAM Sends data to an external script or program A = LOAD ‘data’;
UNION Computes the union of two or more relations. (Does not preserve the order of tuples) X = UNION A, B; A = (1,2,3) B = (2,4)
DUMP X; (4,2,1) (8,9)
(1,2,3) (1,3)


Function Syntax Description
AVG AVG(expression Computes the average of the numeric values in a single-column bag.
CONCAT CONCAT (expression, expression) Concatenates two expressions of identical type.
COUNT COUNT(expression) Computes the number of elements in a bag, it ignores null.
COUNT_STAR COUNT_STAR(expression) Computes the number of elements in a bag, it includes null.
DIFF DIFF (expression, expression) Compares two fields in a tuple, any tuples that are in one bag but not the other are returned in a bag.
DIFF DIFF (expression, expression) Compares two fields in a tuple, any tuples that are in one bag but not the other are returned in a bag.
IsEmpty IsEmpty(expression) Checks if a bag or map is empty.
MAX MAX(expression) Computes the maximum of the numeric values or chararrays in a single-column bag
MIN MIN(expression) Computes the minimum of the numeric values or chararrays in a single-column bag.
SIZE SIZE(expression) Computes the number of elements based on any Pig data type. SIZE includes NULL values in the size computation
SUM SUM(expression) Computes the sum of the numeric values in a single-column bag.
TOKENIZE TOKENIZE(expression [, 'field_delimiter']) Splits a string and outputs a bag of words.

Load/Store Functions

FUnction Syntax Description
Handling Compression A = load ‘myinput.gz’;
store A into ‘myoutput.gz’;
PigStorage and TextLoader support gzip and bzip compression for both read (load) and write (store). BinStorage does not support compression.
BinStorage A = LOAD ‘data’ USING BinStorage(); Loads and stores data in machine-readable format.
JsonLoader, JsonStorage A = load ‘a.json’ using JsonLoader(); Load or store JSON data.
PigDump STORE X INTO ‘output’ USING PigDump(); Stores data in UTF-8 format.
PigStorage A = LOAD ‘student’ USING PigStorage(‘\t’) AS (name: chararray, age:int, gpa: float); Loads and stores data as structured text files.
TextLoader A = LOAD ‘data’ USING TextLoader(); Loads unstructured data in UTF-8 format.

Math Functions

Operator Description Example
ABS ABS(expression) Returns the absolute value of an expression. If the result is not negative (x ≥ 0), the result is returned. If the result is negative (x < 0), the negation of the result is returned.
ACOS ACOS(expression) Returns the arc cosine of an expression.
ASIN ASIN(expression) Returns the arc sine of an expression.
ATAN ATAN(expression) Returns the arc tangent of an expression.
CBRT CBRT(expression) Returns the cube root of an expression.
CEIL CEIL(expression) Returns the value of an expression rounded up to the nearest integer. This function never decreases the result value.
COS COS(expression) Returns the trigonometric cosine of an expression.
COSH COSH(expression) Returns the hyperbolic cosine of an expression.
EXP EXP(expression) Returns Euler’s number e raised to the power of x.
FLOOR FLOOR(expression) Returns the value of an expression rounded down to the nearest integer. This function never increases the result value.
LOG LOG(expression) Returns the natural logarithm (base e) of an expression.
LOG10 LOG10(expression) Returns the base 10 logarithm of an expression.
RANDOM RANDOM( ) Returns a pseudo random number (type double) greater than or equal to 0.0 and less than 1.0.
ROUND ROUND(expression) Returns the value of an expression rounded to an integer (if the result type is float) or rounded to a long (if the result type is double).
SIN SIN(expression) Returns the sine of an expression.
SINH SINH(expression) Returns the hyperbolic sine of an expression.
SQRT SQRT(expression) Returns the positive square root of an expression.
TAN TAN(expression) Returns the trignometric tangent of an angle.
TANH TANH(expression) Returns the hyperbolic tangent of an expression.

String Functions

Operator Description Example
INDEXOF INDEXOF(string, ‘character’, startIndex) Returns the index of the first occurrence of a character in a string, searching forward from a start index.
LAST_INDEX LAST_INDEX_OF(expression) Returns the index of the last occurrence of a character in a string, searching backward from a start index.
LCFIRST LCFIRST(expression) Converts the first character in a string to lower case.
LOWER LOWER(expression) Converts all characters in a string to lower case.
REGEX_EXTRACT REGEX_EXTRACT (string, regex, index) Performs regular expression matching and extracts the matched group defined by an index parameter. The function uses Java regular expression form.
REGEX_EXTRACT_ALL REGEX_EXTRACT (string, regex) Performs regular expression matching and extracts all matched groups. The function uses Java regular expression form.
REPLACE REPLACE(string, ‘oldChar’, ‘newChar’); Replaces existing characters in a string with new characters.
STRSPLIT STRSPLIT(string, regex, limit) Splits a string around matches of a given regular expression.
SUBSTRING SUBSTRING(string, startIndex, stopIndex) Returns a substring from a given string.
TRIM TRIM(expression) Returns a copy of a string with leading and trailing white space removed.
UCFIRST UCFIRST(expression) Returns a string with the first character converted to upper case.
UPPER UPPER(expression) Returns a string converted to upper case.

Tuple, Bag, Map Functions

Operator Description Example
TOTUPLE TOTUPLE(expression [, expression ...]) Converts one or more expressions to type tuple.
TOBAG TOBAG(expression [, expression ...]) Converts one or more expressions to individual tuples which are then placed in a bag.
TOMAP TOMAP(key-expression, value-expression [, key-expression, value-expression ...]) Converts key/value expression pairs into a map. Needs an even number of expressions as parameters. The elements must comply with map type rules.
TOP TOP(topN,column,relation) Returns the top-n tuples from a bag of tuples.

User Defined Functions (UDFs)

Pig provides extensive support for user defined functions (UDFs) as a way to specify custom processing. Pig UDFs can currently be implemented in three languages: Java, Python, JavaScript and Ruby.
Registering UDFs
Registering Java UDFs:
---register_java_udf.pig  register 'your_path_to_piggybank/piggybank.jar';  divs      = load 'NYSE_dividends' as (exchange:chararray, symbol:chararray,                  date:chararray, dividends:float);
Registering Python UDFs (The Python script must be in your current directory):
--register_python_udf.pig  register '' using jython as bballudfs;  players  = load 'baseball' as (name:chararray, team:chararray,                  pos:bag{t:(p:chararray)}, bat:map[]);
Writing UDFs
Java UDFs:
package myudfs;  import;  import org.apache.pig.EvalFunc;  import;    public class UPPER extends EvalFunc  {     public String exec(Tuple input) throws IOException {         if (input == null || input.size() == 0)             return null;             try{                String str = (String)input.get(0);                return str.toUpperCase();             }catch(Exception e){                throw new IOException("Caught exception processing input row ", e);             }        }    }
Python UDFs
#Square - Square of a number of any data type  @outputSchemaFunction("squareSchema") -- Defines a script delegate function that defines schema for this function depending upon the input type.  def square(num):     return ((num)*(num))  @schemaFunction("squareSchema") --Defines delegate function and is not registered to Pig.   def squareSchema(input):     return input     #Percent- Percentage   @outputSchema("percent:double") --Defines schema for a script UDF in a format that Pig understands and is able to parse   def percent(num, total):     return num * 100 / total

Data Types

Simple Types

Operator Description Example
int Signed 32-bit integer 10
long Signed 64-bit integer Data: 10L or 10l
Display: 10L
float 32-bit floating point Data: 10.5F or 10.5f or 10.5e2f or 10.5E2F
Display: 10.5F or 1050.0F
double 64-bit floating point Data: 10.5 or 10.5e2 or 10.5E2
Display: 10.5 or 1050.0
chararray Character array (string) in Unicode UTF-8 format hello world
bytearray Byte array (blob)
boolean boolean true/false (case insensitive)

Wednesday, 19 February 2014

Download Windows 8.1 Retail and OEM .iso

I would be grateful if you leave a comment below to let me know if this guide was easy to follow and worked successfully in particular if you have an OEM license – a system where Windows came preinstalled by Dell or HP or other major vendors. Please also comment on the model and if the system came with Windows 8.0 or Windows 8.1.

Download Windows 8.1 .iso

Microsoft have revised their downloader for Windows 8.1 with Update 1.
Their new downloader allows for the selection of the version of Windows 8.1, the language and the architecture so can be used on any Windows system to prepare installation media which was requested here and many pother places.
To download Windows 8.1 go here:
The direct link to the downloader is here:
Double click the mediacreationtool
Select Run
The downloader will load displaying the Windows logo:
You will asked for Language, Edition and Architecture:
Microsoft offer a large assortment on languages as shown, pick your desired language.
I’m going to select “proper English” i.e. “English – en-gb”:
You will then be prompted for your Windows version – in this case I am going to select Windows 8.1:
For the OEM license its important to note whether your product key is Windows 8.1 Home (often denoted as just Windows 8.1) or Windows 8.1 Professional. If your system was sold with Windows 8.1 or Windows 8.1 Professional stickers like these should be affixed at the top or base of the system stating the version of Windows.
Then you will be prompted for your architecture. I’m going to select the 64 Bit version, in most cases the 64 Bit version should be selected. The 32 Bit version should only be used for legacy applications and on new hardware this is better done in a Virtual Machine.
When you have selected your 3 desired options select next:
Then select to save the .iso file and select next:
Select the location to save and name of your .iso and select next:
The .iso will download:
13 14
15 16
Once you .iso is saved select Finish. If you want to download another version simply launch the media creation tool again.