$ groups simon staff everyone localaccounts _appserverusr admin _appserveradm _lpadmin _appstore _lpoperator _developer _analyticsusers com.apple.access_ftp com.apple.access_screensharing com.apple.access_ssh com.apple.sharepoint.group.1
查询指定用户的信息
id -a <username>
1 2 3
$ id -a simon uid=501(simon) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm), 98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),701(com.apple.sharepoint.group.1)
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
<configuration>
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hivemetastoredb?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property>
<property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property>
<property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>username to use against metastore database</description> </property>
<property> <name>javax.jdo.option.ConnectionPassword</name> <value>PASSWORD</value> <description>password to use against metastore database</description> </property>
ls: /usr/local/spark/spark-2.2.3-bin-hadoop2.6/lib/spark-assembly-*.jar: No such file or directory 2019-01-11 10:12:46,772 WARN [main] mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hbase/hbase-1.2.0-cdh5.7.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2019-01-11 10:12:46,954 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Logging initialized using configuration in jar:file:/usr/local/hive/hive-1.1.0-cdh5.7.0/lib/hive-common-1.1.0-cdh5.7.0.jar!/hive-log4j.properties Thu Jan 11 10:12:49 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Thu Jan 11 10:12:49 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Thu Jan 11 10:12:49 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Thu Jan 11 10:12:49 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Thu Jan 11 10:12:51 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Thu Jan 11 10:12:51 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Thu Jan 11 10:12:51 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Thu Jan 11 10:12:51 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. WARNING: Hive CLI is deprecated and migration to Beeline is recommended. hive>
cd $MYSQL_HOME bin/mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 533 Server version: 5.7.17 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
show databases; +---------------------------+ | Database | +---------------------------+ | information_schema | | hivemetastoredb | | mysql | | performance_schema | +---------------------------+ 11 rows in set (0.01 sec)
cd $HADOOP_HOME bin/hadoop fs -text /data/input 19/01/11 11:00:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hello world hello hello world
我们把hdfs://data/input加载到表中
1 2 3 4 5
hive> LOAD DATA INPATH '/data/input' INTO TABLE hive_wordcount; Loading data to table default.hive_wordcount Table default.hive_wordcount stats: [numFiles=1, totalSize=30] OK Time taken: 1.078 seconds
hive> select word, count(1) from hive_wordcount lateral view explode(split(context, ' ')) wc as word group by word; Query ID = simon_20190111105555_39805c49-890c-4241-8388-56542c90fa2e Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1548765492126_0003, Tracking URL = http://localhost:8088/proxy/application_1548765492126_0003/ Kill Command = /usr/local/hadoop/hadoop-2.6.0-cdh5.7.0/bin/hadoop job -kill job_1548765492126_0003 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2019-01-11 11:11:01,153 Stage-1 map = 0%, reduce = 0% 2019-01-11 11:11:06,299 Stage-1 map = 100%, reduce = 0% 2019-01-11 11:11:12,475 Stage-1 map = 100%, reduce = 100% Ended Job = job_1548765492126_0003 MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 8786 HDFS Write: 16 SUCCESS Total MapReduce CPU Time Spent: 0 msec OK hello 3 world 2 Time taken: 18.997 seconds, Fetched: 2 row(s)
sbin/start-yarn.sh starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-simon-resourcemanager-localhost.out localhost: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-simon-nodemanager-localhost.out
1 2 3
jps 41620 NodeManager 41546 ResourceManager
说明YARN已经成功启动。
通过浏览器访问地址 http://localhost:8088，可见页面如下
提交作业
1 2
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount Usage: wordcount <in> [<in>...] <out>
说明需要提供输入文件及输出结果的位置。
我们先创建一个测试文件
1 2 3 4
vi /tmp/input
hello world hello hello world
把input文件上传到hdfs://data/目录下
1
bin/hadoop fs -put /tmp/input /data
再次执行命令
1
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /data/input /data/out
查看执行结果
1 2 3 4 5
bin/hadoop fs -ls /data/out 19/01/09 21:07:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-r--r-- 1 simon supergroup 0 2019-01-29 21:06 /data/out/_SUCCESS -rw-r--r-- 1 simon supergroup 16 2019-01-29 21:06 /data/out/part-r-00000
wordcount的结果在文件/data/out/part-r-00000里面
1 2 3 4
bin/hadoop fs -text /data/out/part-r-00000 19/01/09 21:08:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hello 3 world 2
如果重复提交作业，会出现如下FileAlreadyExistsException异常。
1 2 3 4 5
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /data/input /data/out 19/01/09 21:09:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/01/09 21:09:53 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 19/01/09 21:09:53 WARN security.UserGroupInformation: PriviledgedActionException as:simon (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:8020/data/out already exists org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:8020/data/out already exists
