{"id":1634,"date":"2021-01-21T19:19:49","date_gmt":"2021-01-21T11:19:49","guid":{"rendered":"https:\/\/www.specialwu.com\/?p=1634"},"modified":"2021-04-12T17:35:12","modified_gmt":"2021-04-12T09:35:12","slug":"%e5%a4%a7%e6%95%b0%e6%8d%ae%e5%ad%98%e5%82%a8%e7%bb%84%e4%bb%b6-hdfs","status":"publish","type":"post","link":"http:\/\/www.specialwu.com\/?p=1634","title":{"rendered":"\u5927\u6570\u636e\u5b58\u50a8\u7ec4\u4ef6&#8211;hdfs"},"content":{"rendered":"<ul>\n<li>\u5b98\u7f51\u7ed9\u51fa\u7684\u4ecb\u7ecd<\/li>\n<\/ul>\n<p>The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. HDFS relaxes a few POSIX requirements to enable streaming access to file system data. HDFS was originally built as infrastructure for the Apache Nutch web search engine project. HDFS is part of the Apache Hadoop Core project<\/p>\n<p><span class=\"text-highlighted-inline\" style=\"background-color: #fffd38;\"> Hadoop\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\uff08HDFS\uff09\u662f\u4e00\u79cd\u65e8\u5728\u5728\u5546\u54c1\u786c\u4ef6\u4e0a\u8fd0\u884c\u7684\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u3002\u5b83\u4e0e\u73b0\u6709\u7684\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u6709\u8bb8\u591a\u76f8\u4f3c\u4e4b\u5904\u3002\u4f46\u662f\uff0c\u4e0e\u5176\u4ed6\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u7684\u533a\u522b\u5f88\u660e\u663e\u3002HDFS\u5177\u6709\u9ad8\u5ea6\u7684\u5bb9\u9519\u80fd\u529b\uff0c\u65e8\u5728\u90e8\u7f72\u5728\u4f4e\u6210\u672c\u786c\u4ef6\u4e0a\u3002HDFS\u63d0\u4f9b\u5bf9\u5e94\u7528\u7a0b\u5e8f\u6570\u636e\u7684\u9ad8\u541e\u5410\u91cf\u8bbf\u95ee\uff0c\u5e76\u4e14\u9002\u7528\u4e8e\u5177\u6709\u5927\u6570\u636e\u96c6\u7684\u5e94\u7528\u7a0b\u5e8f\u3002HDFS\u653e\u5bbd\u4e86\u4e00\u4e9bPOSIX\u8981\u6c42\uff0c\u4ee5\u5b9e\u73b0\u5bf9\u6587\u4ef6\u7cfb\u7edf\u6570\u636e\u7684\u6d41\u5f0f\u8bbf\u95ee\u3002HDFS\u6700\u521d\u662f\u4f5c\u4e3aApache Nutch Web\u641c\u7d22\u5f15\u64ce\u9879\u76ee\u7684\u57fa\u7840\u7ed3\u6784\u800c\u6784\u5efa\u7684\u3002<\/span><\/p>\n<p><a class=\"wp-editor-md-post-content-link\" href=\"https:\/\/hadoop.apache.org\/docs\/stable\/hadoop-project-dist\/hadoop-hdfs\/HdfsDesign.html\" title=\"\u70b9\u51fb\u67e5\u770bHadoop\u5b98\u65b9\u6587\u6863\">\u70b9\u51fb\u67e5\u770bhdfs\u5b98\u65b9\u6587\u6863<\/a><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_61 ez-toc-wrap-center counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title \" >\u76ee\u5f55<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #000000;color:#000000\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #000000;color:#000000\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E5%89%8D%E8%A8%80\" title=\"\u524d\u8a00\">\u524d\u8a00<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%B8%80%E3%80%81%E6%98%AF%E4%BB%80%E4%B9%88\" title=\"\u4e00\u3001\u662f\u4ec0\u4e48\">\u4e00\u3001\u662f\u4ec0\u4e48<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%BA%8C%E3%80%81%E6%80%8E%E4%B9%88%E7%94%A8\" title=\"\u4e8c\u3001\u600e\u4e48\u7528\">\u4e8c\u3001\u600e\u4e48\u7528<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E9%85%8D%E7%BD%AE%E8%BF%87%E7%A8%8B\" title=\"\u914d\u7f6e\u8fc7\u7a0b\">\u914d\u7f6e\u8fc7\u7a0b<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E5%90%84%E4%B8%AA%E6%96%87%E4%BB%B6%E4%B8%8B%E7%9A%84%E5%86%85%E5%AE%B9\" title=\"\u5404\u4e2a\u6587\u4ef6\u4e0b\u7684\u5185\u5bb9\">\u5404\u4e2a\u6587\u4ef6\u4e0b\u7684\u5185\u5bb9<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E5%90%91hdfs%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6%E4%B8%8B%E8%BD%BD%E6%96%87%E4%BB%B6\" title=\"\u5411hdfs\u4e0a\u4f20\u6587\u4ef6\u4e0b\u8f7d\u6587\u4ef6\">\u5411hdfs\u4e0a\u4f20\u6587\u4ef6\u4e0b\u8f7d\u6587\u4ef6<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%BD%BF%E7%94%A8Java%E4%BB%A3%E7%A0%81%E5%90%91hdfs%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\" title=\"\u4f7f\u7528Java\u4ee3\u7801\u5411hdfs\u4e0a\u4f20\u6587\u4ef6\">\u4f7f\u7528Java\u4ee3\u7801\u5411hdfs\u4e0a\u4f20\u6587\u4ef6<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%BD%BF%E7%94%A8hdfs%E5%91%BD%E4%BB%A4%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\" title=\"\u4f7f\u7528hdfs\u547d\u4ee4\u4e0a\u4f20\u6587\u4ef6\">\u4f7f\u7528hdfs\u547d\u4ee4\u4e0a\u4f20\u6587\u4ef6<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%BD%BF%E7%94%A8%E5%85%B6%E4%BB%96%E7%BB%84%E4%BB%B6\" title=\"\u4f7f\u7528\u5176\u4ed6\u7ec4\u4ef6\">\u4f7f\u7528\u5176\u4ed6\u7ec4\u4ef6<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%BD%BF%E7%94%A8sqoop%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\" title=\"\u4f7f\u7528sqoop\u4e0a\u4f20\u6587\u4ef6\">\u4f7f\u7528sqoop\u4e0a\u4f20\u6587\u4ef6<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%BD%BF%E7%94%A8flume%E5%90%91hdfs%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\" title=\"\u4f7f\u7528flume\u5411hdfs\u4e0a\u4f20\u6587\u4ef6\">\u4f7f\u7528flume\u5411hdfs\u4e0a\u4f20\u6587\u4ef6<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%BD%BF%E7%94%A8hive%E5%B0%86%E6%95%B0%E6%8D%AE%E5%AF%BC%E5%88%B0hdfs%E4%B8%8A\" title=\"\u4f7f\u7528hive\u5c06\u6570\u636e\u5bfc\u5230hdfs\u4e0a\">\u4f7f\u7528hive\u5c06\u6570\u636e\u5bfc\u5230hdfs\u4e0a<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%B8%89%E3%80%81hdfs%E6%A0%B8%E5%BF%83%E9%97%AE%E9%A2%98\" title=\"\u4e09\u3001hdfs\u6838\u5fc3\u95ee\u9898\">\u4e09\u3001hdfs\u6838\u5fc3\u95ee\u9898<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#hdfs%E5%9B%9B%E5%A4%A7%E6%9C%BA%E5%88%B6\" title=\"hdfs\u56db\u5927\u673a\u5236\">hdfs\u56db\u5927\u673a\u5236<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#HDFS_HA%E6%A8%A1%E5%BC%8F%EF%BC%88%E9%AB%98%E5%8F%AF%E7%94%A8%E6%A8%A1%E5%BC%8F%EF%BC%89\" title=\"HDFS HA\u6a21\u5f0f\uff08\u9ad8\u53ef\u7528\u6a21\u5f0f\uff09\">HDFS HA\u6a21\u5f0f\uff08\u9ad8\u53ef\u7528\u6a21\u5f0f\uff09<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#%E4%B8%BA%E4%BA%86%E6%9E%84%E5%BB%BAHA%E9%9B%86%E7%BE%A4%E6%9E%B6%E6%9E%84%EF%BC%8C%E4%BD%A0%E9%9C%80%E8%A6%81%E5%87%86%E5%A4%87%E5%A6%82%E4%B8%8B%E8%B5%84%E6%BA%90%EF%BC%9A\" title=\"\u4e3a\u4e86\u6784\u5efaHA\u96c6\u7fa4\u67b6\u6784\uff0c\u4f60\u9700\u8981\u51c6\u5907\u5982\u4e0b\u8d44\u6e90\uff1a\">\u4e3a\u4e86\u6784\u5efaHA\u96c6\u7fa4\u67b6\u6784\uff0c\u4f60\u9700\u8981\u51c6\u5907\u5982\u4e0b\u8d44\u6e90\uff1a<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"http:\/\/www.specialwu.com\/?p=1634\/#hadoop%E8%87%AA%E5%B8%A6%E7%9A%84%E8%A7%A3%E5%86%B3%E5%B0%8F%E6%96%87%E4%BB%B6%E9%97%AE%E9%A2%98%E7%9A%84%E6%96%B9%E6%A1%88\" title=\"hadoop\u81ea\u5e26\u7684\u89e3\u51b3\u5c0f\u6587\u4ef6\u95ee\u9898\u7684\u65b9\u6848\">hadoop\u81ea\u5e26\u7684\u89e3\u51b3\u5c0f\u6587\u4ef6\u95ee\u9898\u7684\u65b9\u6848<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"%E5%89%8D%E8%A8%80\"><\/span>\u524d\u8a00<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<p>\u4e00\u63d0\u8d77\u5927\u6570\u636e\uff0c\u9996\u5148\u60f3\u5230\u7684\u662f\u5b83\u6d77\u91cf\u7684\u6570\u636e\uff0c\u6570\u636e\u5b58\u7684\u4e0b\uff0c\u4e0d\u4e22\u5931\uff0c\u65b9\u4fbf\u7ba1\u7406\u662f\u5927\u6570\u636e\u5b58\u50a8\u7ec4\u4ef6\u6700\u57fa\u672c\u7684\u8981\u6c42\uff0c\u672c\u7bc7\u57fa\u4e8e\u8fd9\u4e9b\u770b\u4f3c\u57fa\u672c\u7684\u8981\u6c42\u5c55\u5f00\u3002<br \/>\n\u5047\u8bbe\u6211\u4eec\u67091T\u7684\u6570\u636e\u9700\u8981\u5b58\u50a8\uff0c\u90a3\u4e48\u6309\u7167\u5e38\u89c4\u624b\u6bb5\u6211\u4eec\u628a\u8fd9\u6574\u4e2a\u6570\u636e\u5b58\u5728\u4e00\u53f0\u673a\u5668\u4e0a\u8fdb\u884c\u5b58\u50a8\uff0c\u5982\u679c\u6211\u4eec\u9700\u8981\u627e\u5230\u67d0\u4e00\u4e2a\u6587\u4ef6\u90a3\u4e48\u5c31\u9700\u8981\u5f88\u957f\u7684\u5bfb\u5740\u65f6\u95f4\uff0c\u90a3\u4e48\u6309\u7167\u73b0\u5728\u7684\u5927\u6570\u636e\u5b58\u50a8\u6280\u672f\uff0c\u6211\u4eec\u5c06\u8fd91T\u7684\u6570\u636e\u5206\u522b\u653e\u5728100\u53f0\u673a\u5668\u4e0a\uff0c\u90a3\u4e48\u5bfb\u5740\u65f6\u95f4\u5927\u5927\u51cf\u5c11\uff0c\u90a3\u4e48\u4f60\u4f1a\u60f3\u201c\u6bcf\u53f0\u673a\u5668\u53ea\u653e\u8fd9\u4e00\u90e8\u5206\u7684\u6570\u636e\u80af\u5b9a\u662f\u6d6a\u8d39\u7a7a\u95f4\u4e86\u201d\uff0c\u90a3\u4e48hdfs\u5c06\u5269\u4f59\u90e8\u5206\u4e5f\u540c\u6837\u5b58\u653e\u4e00\u6574\u4e2a\u6587\u4ef6\u7684\u90e8\u5206\u6570\u636e\uff0c\u8fd9\u4e5f\u5c31\u7275\u626f\u51fahdfs\u5757\u7684\u6982\u5ff5\uff0c\u8fd9\u6837\u53c8\u7275\u626f\u51fa\u4e86\u4e00\u4e2a\u95ee\u9898\uff0c\u5b58\u653e\u6574\u4e2a\u6570\u636e\u7684\u8282\u70b9\u589e\u52a0\u4e86\uff0c\u90a3\u4e48\u96be\u514d\u51fa\u73b0\u4e00\u4e2a\u8282\u70b9\u51fa\u73b0\u6545\u969c\u5bfc\u81f4\u6570\u636e\u4e22\u5931\u7684\u60c5\u51b5\u7b49\u7b49\u4e00\u4e9b\u5217\u95ee\u9898\u3002<br \/>\n\u672c\u7bc7\u5148\u5efa\u7acb\u4e00\u4e9b\u7ec4\u4ef6\u7684\u6982\u5ff5\uff0c\u7136\u540e\u518d\u56f4\u7ed5\u8fd9\u4e9b\u95ee\u9898\u5c55\u5f00\u3002\u672c\u7bc7\u6587\u7ae0\u7ed3\u6784\uff1a<br \/>\n1\u3001\u662f\u4ec0\u4e48\uff0c\u6709\u4ec0\u4e48\u7279\u70b9\uff1b\uff08fsimage,edits\u6587\u4ef6\u957f\u4ec0\u4e48\u6837\u5b50\uff09<br \/>\n2\u3001\u6709\u4ec0\u4e48\u4f5c\u7528\uff0c\u5b83\u8be5\u600e\u4e48\u7528\uff1b\uff08\u914d\u7f6e\u8fc7\u7a0b\uff0c\u4e0a\u4f20\u6587\u4ef6\uff09<br \/>\n3\u3001\u5b83\u7684\u6838\u5fc3\u95ee\u9898\uff1b\uff08HA\u6a21\u5f0f\uff0c\u56db\u5927\u673a\u5236\uff0c\u8d1f\u8f7d\u5747\u8861\uff0c\u5c0f\u6587\u4ef6\u95ee\u9898\uff09<\/p>\n<h1><span class=\"ez-toc-section\" id=\"%E4%B8%80%E3%80%81%E6%98%AF%E4%BB%80%E4%B9%88\"><\/span>\u4e00\u3001\u662f\u4ec0\u4e48<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<ul>\n<li>\u80cc\u666f<\/li>\n<\/ul>\n<p>\u968f\u7740\u6570\u636e\u91cf\u8d8a\u6765\u8d8a\u5927\uff0c\u5728\u4e00\u4e2a\u64cd\u4f5c\u7cfb\u7edf\u7ba1\u8f96\u7684\u8303\u56f4\u5185\u5b58\u4e0d\u4e0b\u4e86\uff0c\u90a3\u4e48\u5c31\u5206\u914d\u5230\u66f4\u591a\u7684\u64cd\u4f5c\u7cfb\u7edf\u7ba1\u7406\u7684\u78c1\u76d8\u4e2d\uff0c\u4f46\u662f\u4e0d\u65b9\u4fbf\u7ba1\u7406\u548c\u7ef4\u62a4\uff0c\u8feb\u5207\u9700\u8981\u4e00\u79cd\u7cfb\u7edf\u6765\u7ba1\u7406\u591a\u53f0\u673a\u5668\u4e0a\u7684\u6587\u4ef6\uff0c\u8fd9\u5c31\u662f\u5206\u5e03\u5f0f\u6587\u4ef6\u7ba1\u7406\u7cfb\u7edf\u3002The Hadoop Distributed File System (HDFS)\uff1b<\/p>\n<ul>\n<li>\u662f\u4ec0\u4e48\/\u6709\u4ec0\u4e48\u7279\u70b9<\/li>\n<\/ul>\n<ol>\n<li>\u63d0\u4f9b\u5bf9\u5e94\u7528\u7a0b\u5e8f\u6570\u636e\u7684\u9ad8\u541e\u5410\u91cf\u8bbf\u95ee\u7684\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u3002\u53ef\u5b58\u50a8PB\u7ea7\u522b\u6570\u636e<\/li>\n<li>HDFS\u5177\u6709\u5f88\u9ad8\u7684\u5bb9\u9519\u80fd\u529b\uff0c\u88ab\u8bbe\u8ba1\u4e3a\u90e8\u7f72\u5728\u4f4e\u6210\u672c\u7684\u786c\u4ef6\u4e0a\u3002HDFS\u63d0\u4f9b\u4e86\u5bf9\u5e94\u7528\u7a0b\u5e8f\u6570\u636e\u7684\u9ad8\u541e\u5410\u91cf\u8bbf\u95ee\uff0c\u9002\u7528\u4e8e\u5177\u6709\u5927\u578b\u6570\u636e\u96c6\u7684\u5e94\u7528\u7a0b\u5e8f\u3002<\/li>\n<li>HDFS\u653e\u677e\u4e86\u7b80\u4fbf\u64cd\u4f5c\u7cfb\u7edf\u7684\u4e00\u4e9b\u8981\u6c42\uff0c\u4ee5\u652f\u6301\u5bf9\u6587\u4ef6\u7cfb\u7edf\u6570\u636e\u7684\u6d41\u8bbf\u95ee\u3002HDFS\u66f4\u591a\u7684\u662f\u4e3a\u6279\u5904\u7406\u800c\u8bbe\u8ba1\uff0c\u800c\u4e0d\u662f\u7528\u6237\u4ea4\u4e92\u4f7f\u7528\u3002\u91cd\u70b9\u662f\u6570\u636e\u8bbf\u95ee\u7684\u9ad8\u541e\u5410\u91cf\uff0c\u800c\u4e0d\u662f\u4f4e\u5ef6\u8fdf\u7684\u6570\u636e\u8bbf\u95ee\u3002\u5bf9\u4e8e\u6279\u5904\u7406\u7684\u8bbe\u8ba1\u800c\u8a00\u5bfb\u5740\u65f6\u95f4\u662f\u5b83\u7684\u4e00\u4e2a\u74f6\u9888<\/li>\n<li>\u6587\u4ef6\u4e00\u65e6\u521b\u5efa\u3001\u5199\u5165\u548c\u5173\u95ed\uff0c\u9664\u4e86\u8ffd\u52a0\u548c\u622a\u65ad\u4e4b\u5916\uff0c\u4e0d\u9700\u8981\u66f4\u6539\u3002\u652f\u6301\u5c06\u5185\u5bb9\u9644\u52a0\u5230\u6587\u4ef6\u672b\u5c3e\uff0c\u4f46\u4e0d\u80fd\u5728\u4efb\u610f\u70b9\u66f4\u65b0\u3002\u8fd9\u79cd\u5047\u8bbe\u7b80\u5316\u4e86\u6570\u636e\u4e00\u81f4\u6027\u95ee\u9898\uff0c\u5e76\u652f\u6301\u9ad8\u541e\u5410\u91cf\u6570\u636e\u8bbf\u95ee\u3002MapReduce\u5e94\u7528\u7a0b\u5e8f\u6216\u7f51\u7edc\u722c\u866b\u5e94\u7528\u7a0b\u5e8f\u975e\u5e38\u9002\u5408\u8fd9\u4e2a\u6a21\u578b\u3002<\/li>\n<li>\u65e0\u6cd5\u9ad8\u6548\u7684\u5bf9\u5927\u91cf\u5c0f\u6587\u4ef6\u8fdb\u884c\u5b58\u50a8<br \/>\n5.1 \u5b58\u50a8\u5927\u91cf\u5c0f\u6587\u4ef6\u7684\u8bdd\uff0c\u5b83\u4f1a\u5360\u7528 NameNode\u5927\u91cf\u7684\u5185\u5b58\u6765\u5b58\u50a8\u6587\u4ef6\u3001\u76ee\u5f55\u548c\u5757\u4fe1\u606f\u3002\u8fd9\u6837\u662f\u4e0d\u53ef\u53d6\u7684\uff0c\u56e0\u4e3aNameNode\u7684\u5185\u5b58\u603b\u662f\u6709\u9650\u7684\u3002\uff08\u4e00\u4e2a\u6587\u4ef6\u5927\u6982\u5360150\u5b57\u8282\u7684\u5185\u5b58\uff09<br \/>\n5.2 \u5c0f\u6587\u4ef6\u5b58\u50a8\u7684\u5bfb\u9053\u65f6\u95f4\u4f1a\u8d85\u8fc7\u8bfb\u53d6\u65f6\u95f4\uff0c\u5b83\u8fdd\u53cd\u4e86HDFS\u7684\u8bbe\u8ba1\u76ee\u6807\u3002<\/li>\n<\/ol>\n<ul>\n<li>\u67b6\u6784<\/li>\n<\/ul>\n<p>\u5411hdfs\u96c6\u7fa4\u4e0a\u4f20\u4e00\u4e2a256M\u5927\u7684\u6587\u4ef6\uff0c\u6574\u4e2a\u6d41\u7a0b<br \/>\n<img src=\"https:\/\/img-blog.csdnimg.cn\/20201027233618869.jpg\" alt=\"\" \/><\/p>\n<ul>\n<li>\u6587\u5b57\u5f52\u7eb3\u5982\u4e0b<\/li>\n<\/ul>\n<p>(1) Client\u5411NameNode\u8282\u70b9\u53d1\u51fa\u8bf7\u6c42;<br \/>\n(2) NameNode\u5411Client\u8fd4\u56de\u53ef\u4ee5\u5b58\u6570\u636e\u7684DataNode\u5217\u8868host1,host2,host3\uff1b<br \/>\n(3) clint\u5c06\u8be5\u6587\u4ef6\u5206\u6210\u4e24\u5757\uff0c\u5927\u5c0f\u5206\u522b\u4e3ablock1\uff0cblock2\uff1b<br \/>\n(4) \u5ba2\u6237\u7aef\u5411wq1\u53d1\u9001block1\uff0c\u53d1\u9001\u8fc7\u7a0b\u662f\u6d41\u5f0f\u5199\u5165\uff0c\u6b64\u65f6\u9ed8\u8ba4\u526f\u672c\u6570\u4e3a2,\u5219host1\u628ablock1\u53d1\u9001\u7ed9host2\uff0chost2\u4f20\u7ed9host3\uff1b<br \/>\n(5) dataNode\u544a\u8bc9client\u81ea\u5df1block1\u5df2\u7ecf\u4f20\u5b8c\uff0c\u540c\u65f6\u8fd9\u4e09\u4e2adatanode\u8282\u70b9\u544a\u8bc9NameNode\u81ea\u5df1\u4f20\u5b8c\u4e86block1\uff1b<br \/>\n(6) block2\u6309\u7167\u540c\u6837\u7684\u65b9\u5f0f\u5b8c\u6210\u6570\u636e\u7684\u4f20\u9012.<\/p>\n<ul>\n<li>\u5404\u4e2a\u90e8\u5206\u7684\u542b\u4e49<\/li>\n<\/ul>\n<ol>\n<li>Client\uff1a\u5ba2\u6237\u7aef\u3002\u5982\u679c\u4f7f\u7528JavaAPI\u5bf9\u96c6\u7fa4\u64cd\u4f5c\u90a3\u4e48\u8fd9\u53f0\u7535\u8111\u5c31\u662f\u5ba2\u6237\u7aef\u3002\n<p>1.1\u6587\u4ef6\u5207\u5206\u3002\u6587\u4ef6\u4e0a\u4f20 HDFS \u7684\u65f6\u5019\uff0cClient \u5c06\u6587\u4ef6\u5207\u5206\u6210\u4e00\u4e2a\u4e00\u4e2a\u7684Block\uff0c\u7136\u540e\u8fdb\u884c\u5b58\u50a8\u3002<br \/>\n1.2\u4e0eNameNode\u4ea4\u4e92\uff0c\u83b7\u53d6\u6587\u4ef6\u7684\u4f4d\u7f6e\u4fe1\u606f\u3002<br \/>\n1.3\u4e0eDataNode\u4ea4\u4e92\uff0c\u8bfb\u53d6\u6216\u8005\u5199\u5165\u6570\u636e\u3002<br \/>\n1.4Client\u63d0\u4f9b\u4e00\u4e9b\u547d\u4ee4\u6765\u7ba1\u7406HDFS\uff0c\u6bd4\u5982\u542f\u52a8\u6216\u8005\u5173\u95edHDFS\u3002<br \/>\n1.5Client\u53ef\u4ee5\u901a\u8fc7\u4e00\u4e9b\u547d\u4ee4\u6765\u8bbf\u95eeHDFS\u3002<\/p>\n<\/li>\n<li>\n<p>NameNode\uff1amaster\uff0cHDFS\u7684\u7ba1\u7406\u8005\u3002<\/p>\n<p>2.1\u7ba1\u7406HDFS\u7684\u540d\u79f0\u7a7a\u95f4\u3002<br \/>\n2.2\u7ba1\u7406\u6570\u636e\u5757\uff08Block\uff09\u6620\u5c04\u4fe1\u606f<br \/>\n2.3\u914d\u7f6e\u526f\u672c\u7b56\u7565<br \/>\n2.4\u5904\u7406\u5ba2\u6237\u7aef\u8bfb\u5199\u8bf7\u6c42\u3002<\/p>\n<\/li>\n<\/ol>\n<p>NameNode \u5728\u542f\u52a8\u7684\u65f6\u5019\u4f1a\u505a\u54ea\u4e9b\u64cd\u4f5c<br \/>\n\u4ece fsimage \u4e2d\u4e0d\u505c\u7684\u987a\u5e8f\u8bfb\u53d6name\u4e0b\u7684\u6587\u4ef6\u548c\u5143\u6570\u636e\u4fe1\u606f\uff0c\u5e76\u5728\u5185\u5b58\u4e2d\u6784\u5efa\u6574\u4e2a namespace\uff0c\u540c\u65f6\u5c06\u6bcf\u4e2a\u6587\u4ef6\u5bf9\u5e94\u7684blockid \u4fdd\u5b58\u5165 BlocksMap \u4e2d\uff0c\u6b64\u65f6 BlocksMap \u4e2d\u6bcf\u4e2a block \u5bf9\u5e94\u7684 datanodes \u5217\u8868\u6682\u65f6\u4e3a\u7a7a\u3002<br \/>\n\u5f53 fsimage \u52a0\u8f7d\u5b8c\u6bd5\u540e\uff0c\u6574\u4e2a HDFS \u7684\u76ee\u5f55\u7ed3\u6784\u5728\u5185\u5b58\u4e2d\u5c31\u5df2\u7ecf\u521d\u59cb\u5316\u5b8c\u6bd5\uff0c\u6240\u7f3a\u7684\u5c31\u662f\u6bcf\u4e2a\u6587\u4ef6\u5bf9\u5e94\u7684 block \u5bf9\u5e94\u7684 datanode \u5217\u8868\u4fe1\u606f\u3002\u8fd9\u4e9b\u4fe1\u606f\u9700\u8981\u4ece datanode \u7684 blockReport \u4e2d\u83b7\u53d6\uff0c\u6240\u4ee5\u52a0\u8f7d fsimage \u5b8c\u6bd5\u540e\uff0cnamenode \u8fdb\u7a0b\u8fdb\u5165 rpc \u7b49\u5f85\u72b6\u6001\uff0c\u7b49\u5f85\u6240\u6709\u7684 datanodes\u53d1\u9001 blockReports<\/p>\n<ol start=\"3\">\n<li>DataNode\uff1aSlave\u3002NameNode\u4e0b\u8fbe\u547d\u4ee4\uff0cDataNode\u6267\u884c\u5b9e\u9645\u7684\u64cd\u4f5c\u3002\n<p>3.1\u5b58\u50a8\u5b9e\u9645\u7684\u6570\u636e\u5757\u3002<br \/>\n3.2\u6267\u884c\u6570\u636e\u5757\u7684\u8bfb\/\u5199\u64cd\u4f5c\u3002<\/p>\n<\/li>\n<li>\n<p>Secondary NameNode\uff1a\u5e76\u975eNameNode\u7684\u70ed\u5907(\u5f53NameNode\u6302\u6389\u7684\u65f6\u5019\uff0c\u5b83\u5e76\u4e0d\u80fd\u9a6c\u4e0a\u66ff\u6362NameNode\u5e76\u63d0\u4f9b\u670d\u52a1\u3002)<\/p>\n<p>4.1\u8f85\u52a9NameNode\uff0c\u5206\u62c5\u5176\u5de5\u4f5c\u91cf,\u968f\u7740edits\u6587\u4ef6\u5728\u64cd\u4f5c\u4e2d\u4f1a\u8d8a\u6765\u8d8a\u5927\uff0c\u6240\u4ee5\u9700\u8981\u63d0\u524d\u5408\u5e76\u5904\u7406\u52a0\u5165\u5185\u5b58<br \/>\n4.2\u5b9a\u671f\u5408\u5e76Fsimage\u548cEdits\uff0c\u5e76\u63a8\u9001\u7ed9NameNode\u3002<br \/>\n4.3\u5728\u7d27\u6025\u60c5\u51b5\u4e0b\uff0c\u53ef\u8f85\u52a9\u6062\u590dNameNode\u3002<\/p>\n<\/li>\n<\/ol>\n<pre data-language=XML><code class=\"language-markup line-numbers\">#1\u4e2a\u5c0f\u65f6\u5408\u5e76\u4e00\u6b21\n&lt;property&gt;\n    &lt;name&gt;fs.checkpoint.period&lt;\/name&gt;\n    &lt;value&gt;3600&lt;\/value&gt;\n&lt;\/property&gt;\n#\u5f53EditsLog\u6587\u4ef6\u5927\u5c0f\u8d85\u8fc764MB\u5408\u5e76\u4e00\u6b21\n&lt;property&gt;\n    &lt;name&gt;fs.checkpoint.size&lt;\/name&gt;\n    &lt;value&gt;67108864&lt;\/value&gt;\n&lt;\/property&gt;\n&lt;property&gt;\n  &lt;name&gt;dfs.namenode.checkpoint.check.period&lt;\/name&gt;\n  &lt;value&gt;60&lt;\/value&gt;\n&lt;description&gt; 1\u5206\u949f\u68c0\u67e5\u4e00\u6b21\u64cd\u4f5c\u6b21\u6570&lt;\/description&gt;\n&lt;\/property &gt;\n#\u56de\u6536\u7ad9\n&lt;property&gt;\n      &lt;name&gt;fs.trash.interval&lt;\/name&gt;\n      &lt;value&gt;120&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<blockquote>\n<p>\u4e3a\u4ec0\u4e48HDFS\u6587\u4ef6\u5757\uff08block\uff09\u5927\u5c0f\u8bbe\u5b9a\u4e3a128M\n<\/p><\/blockquote>\n<p>\u4e0a\u4f20\u540c\u6837\u5927\u5c0f\u7684\u6587\u4ef6\uff0c\u53ef\u51fa\u73b0\u4ee5\u4e0b\u4e24\u79cd\u60c5\u5f62\u3002<br \/>\n\u6587\u4ef6\u5757\u5927\uff0c\u5bfb\u5740\u65f6\u95f4\u77ed\uff0c\u78c1\u76d8\u4f20\u8f93\u65f6\u95f4\u957f<br \/>\n\u6587\u4ef6\u5757\u5c0f\uff0c\u5bfb\u5740\u65f6\u95f4\u957f\uff0c\u78c1\u76d8\u4f20\u8f93\u65f6\u95f4\u77ed<br \/>\n\u5bfb\u5740\u65f6\u95f4\u4e3a\u4f20\u8f93\u65f6\u95f4\u76841%\u65f6\uff0c\u4e3a\u6700\u4f73\u72b6\u6001\uff0c\u56e0\u4e3a\u96c6\u7fa4\u5728\u4f7f\u7528\u8fc7\u7a0b\u4e2d\u662f\u5343\u5146\u7f51\u7edc\u4f20\u8f93\uff0c\u5219\u7f51\u901f\u5927\u7ea6\u6bcf\u79d2100M\uff0cHDFS\u7684\u5e73\u5747\u5bfb\u5740\u65f6\u95f4\u4e3a10ms,10ms\/0.01=1000ms=1s,1sX100M<\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">&lt;property&gt;\u00a0\u00a0\n&lt;name&gt;dfs.blocksize&lt;\/name&gt;\u00a0\u00a0\n&lt;value&gt;30m&lt;\/value&gt;\u00a0\u00a0\n&lt;\/property&gt;\n<\/code><\/pre>\n<h1><span class=\"ez-toc-section\" id=\"%E4%BA%8C%E3%80%81%E6%80%8E%E4%B9%88%E7%94%A8\"><\/span>\u4e8c\u3001\u600e\u4e48\u7528<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<p>\u77e5\u9053\u600e\u4e48\u7528\u7684\u5148\u660e\u786e\u4e0b\u662f\u600e\u4e48\u914d\u7f6e\u7684<\/p>\n<h2><span class=\"ez-toc-section\" id=\"%E9%85%8D%E7%BD%AE%E8%BF%87%E7%A8%8B\"><\/span>\u914d\u7f6e\u8fc7\u7a0b<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>\u7b2c\u4e00\u6b65\uff1a\u5b89\u88c5\u51e0\u53f0CentOS7\u7cfb\u7edf\u7684\u4e3b\u673a<br \/>\n\u7b2c\u4e8c\u6b65\uff1a\u5efa\u7acb\u4e3b\u673a\u540d\u4e0eIP\u5730\u5740\u7684\u6620\u5c04<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 etc]# cat hosts\n127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\n::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\n\u4f60\u7684IP\u5730\u5740       \u4f60\u60f3\u5199\u7684\u4e3b\u673a\u540d\n192.168.111.111  myname\n<\/code><\/pre>\n<p>\u7b2c\u4e09\u6b65\uff1a\u914d\u7f6eSSH\u514d\u5bc6\u767b\u5f55\uff0c\u8fd9\u6b65\u9002\u5408\u5728\u81ea\u5df1\u865a\u62df\u673a\u4e0a\u642d\u5efa\u64cd\u4f5c\uff0c\u751f\u4ea7\u4e2d\u514d\u5bc6\u767b\u9646\u8fd8\u662f\u4e0d\u5b89\u5168\uff0c\u6b64\u6b65\u53ef\u4e0d\u7528\u64cd\u4f5c<\/p>\n<pre><code class=\"language-bash line-numbers\">#\u6709\u51e0\u53f0\u673a\u5668\u6267\u884c\u51e0\u6b21\u8fd9\u51e0\u4e2a\u547d\u4ee4\n1\u3001\u628a\u8001\u7684\u9501\u548c\u94a5\u5319\u90fd\u6254\u4e86\nrm -rf \/root\/.ssh\n2\u3001\u914d\u94a5\u5319\n\u96c6\u7fa4\u4e2d\u7684\u6240\u6709\u673a\u5668\u6267\u884c\uff0c\u8f93\u5165\u8be5\u547d\u4ee4\u540e\u70b9\u51fb\u56de\u8f66\u4e09\u6b21\nssh-keygen -t rsa\n3\u3001\u53d1\u94a5\u5319\n\u4e09\u53f0\u673a\u5668\u90fd\u8981\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\n# \u5c06\u94a5\u5319\u590d\u5236\u7ed9wq1\nssh-copy-id wq1\n# \u5c06\u94a5\u5319\u590d\u5236\u7ed9wq2\nssh-copy-id wq2\n# \u5c06\u94a5\u5319\u590d\u5236\u7ed9wq3\nssh-copy-id wq3\n<\/code><\/pre>\n<p>\u5982\u679c\u4e0d\u505a\u514d\u5bc6\u8bbe\u7f6e\u5c06\u4f1a\u662f\u8fd9\u79cd\u60c5\u51b5(\u5c31\u9700\u8981\u8f93\u5165\u591a\u6b21\u5bc6\u7801\uff0c\u76f8\u5bf9\u9ebb\u70e6\uff0c\u56e0\u6b64\u6709\u4e86\u4e0a\u6b65\u7684\u64cd\u4f5c)<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 ~]# start-dfs.sh\nStarting namenodes on [wq1]\nroot@wq1's password: \nwq1: starting namenode, logging to \/opt\/hadoop-2.7.7\/logs\/hadoop-root-namenode-wq1.out\nroot@wq1's password: \nwq1: starting datanode, logging to \/opt\/hadoop-2.7.7\/logs\/hadoop-root-datanode-wq1.out\nStarting secondary namenodes [wq1]\nroot@wq1's password: \nwq1: starting secondarynamenode, logging to \/opt\/hadoop-2.7.7\/logs\/hadoop-root-secondarynamenode-wq1.out\n[root@wq1 ~]# jps\n3316 NameNode\n3670 SecondaryNameNode\n3464 DataNode\n3806 Jps\n<\/code><\/pre>\n<p>\u7b2c\u56db\u6b65\uff1a\u914d\u7f6e\u73af\u5883\u53d8\u91cf\uff08\u914d\u7f6eJDK\u73af\u5883\uff09<br \/>\n\u7b2c\u4e94\u6b65\uff1a\u4fee\u6539hdfs\u76f8\u5173\u7684\u53c2\u6570<br \/>\n<code>core-site.xml<\/code>(\u914d\u7f6e\u786e\u5b9a\u90a3\u53f0\u673a\u5668\u662fnamenode,\u786e\u5b9a\u4e0a\u4f20\u6587\u4ef6\u5b58\u50a8\u7684\u4f4d\u7f6e)<br \/>\n<code>hdfs-site.xml<\/code>(\u914d\u7f6e\u786e\u5b9a\u90a3\u53f0\u673a\u5668\u662fSecondaryNamenode)<br \/>\n<code>hadoop-env.sh<\/code>(\u5bfc\u5165JDK\u73af\u5883)<br \/>\n<code>slaves<\/code>(\u6240\u6709\u673a\u5668\u540d\u5199\u5165\uff0c\u5f62\u6210\u4e3b\u4ece\u5173\u7cfb)<br \/>\n<code>vim hadoop-env.sh<\/code><\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">#hadoop-env.sh\nexport JAVA_HOME=\/opt\/jdk1.8\n<\/code><\/pre>\n<p><code>vim core-site.xml<\/code><\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">&lt;property&gt;\n    &lt;!-- (\u914d\u7f6eNameNode\u670d\u52a1\u542f\u52a8\u7684\u4e3b\u673a\u548c\u7aef\u53e3) --&gt;\n    &lt;name&gt;fs.defaultFS&lt;\/name&gt;\n    &lt;value&gt;hdfs:\/\/wq1:9000\/&lt;\/value&gt;\n&lt;\/property&gt;\n&lt;property&gt;\n    &lt;!-- \u914d\u7f6ehdfs\u6587\u4ef6\u4fdd\u5b58\u7684\u76ee\u5f55\uff0c\u5f53\u524d\u9009\u62e9\u653e\u5728Hadoop\u7684\u5b89\u88c5\u76ee\u5f55\u4e0b --&gt;\n    &lt;name&gt;hadoop.tmp.dir&lt;\/name&gt;\n    &lt;value&gt;\/opt\/hadoop-2.7.7\/data&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<p><code>vim hdfs-site.xml<\/code><\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">&lt;property&gt;\n    &lt;!-- \u914d\u7f6e\u56fa\u5b9a\u7684\u8282\u70b9\u542f\u52a8SecondaryNamenode\u4e0enamenode\u4e0d\u5728\u540c\u4e00\u8282\u70b9\u5373\u53ef --&gt;\n    &lt;name&gt;dfs.namenode.secondary.http-address&lt;\/name&gt;\n    &lt;value&gt;wq2:50090&lt;\/value&gt;\n&lt;\/property&gt;\n#\u9ad8\u53ef\u7528\u914d\u7f6e\n&lt;property&gt;\n   &lt;name&gt;ha.zookeeper.quorum&lt;\/name&gt;\n   &lt;value&gt;wq1:2181,wq2:2181,wq3:2181&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<p><code>vim slaves<\/code>(\u5199\u5165\u96c6\u7fa4\u4e2d\u7684\u673a\u5668\u540d)<\/p>\n<pre><code class=\"language-bash line-numbers\">wq1\nwq2\n<\/code><\/pre>\n<p>\u7b2c\u516d\u6b65\uff1a\u683c\u5f0f\u5316\uff08\u521d\u59cb\u5316\u4e86\u65b0\u7684\u96c6\u7fa4id\uff0c\u547d\u540d\u7a7a\u95f4\uff09<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 ~]# hdfs namenode -format\n<\/code><\/pre>\n<p>\u7b2c\u4e03\u6b65\uff1a\u542f\u52a8<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 ~]# start-dfs.sh\n<\/code><\/pre>\n<p>\u7b2c\u516b\u6b65\uff1a\u67e5\u770b\u662f\u5426\u542f\u52a8\u8d77\u6765\uff08\u770bJava\u5b88\u62a4\u8fdb\u7a0b\uff09<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 ~]# jps\n7539 DataNode\n7431 NameNode\n28953 Jps\n<\/code><\/pre>\n<h2><span class=\"ez-toc-section\" id=\"%E5%90%84%E4%B8%AA%E6%96%87%E4%BB%B6%E4%B8%8B%E7%9A%84%E5%86%85%E5%AE%B9\"><\/span>\u5404\u4e2a\u6587\u4ef6\u4e0b\u7684\u5185\u5bb9<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li>1\u5b89\u88c5\u76ee\u5f55\u4e0b\u7684\u4e1c\u897f<\/li>\n<\/ul>\n<p>\u5b89\u88c5\u597d\u540e\u5e76\u4e14\u80fd\u8fd0\u884c\u4e4b\u540e\u67e5\u770b\u81ea\u5df1\u7684\u5b89\u88c5\u76ee\u5f55\u4e0b\u90fd\u6709\u54ea\u4e9b\u6587\u4ef6<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq2 hadoop-2.7.7]# pwd\n\/opt\/hadoop-2.7.7\n[root@wq2 hadoop-2.7.7]# ll\n\u603b\u7528\u91cf 116\ndrwxr-xr-x. 2 wuqi2020 ftp    194 7\u6708  19 2018 bin\ndrwxr-xr-x  4 root     root    37 10\u6708 31 18:04 data\ndrwxr-xr-x. 3 wuqi2020 ftp     20 7\u6708  19 2018 etc\ndrwxr-xr-x. 2 wuqi2020 ftp    106 7\u6708  19 2018 include\ndrwxr-xr-x. 3 wuqi2020 ftp     20 7\u6708  19 2018 lib\ndrwxr-xr-x. 2 wuqi2020 ftp    239 7\u6708  19 2018 libexec\n-rw-r--r--. 1 wuqi2020 ftp  86424 7\u6708  19 2018 LICENSE.txt\ndrwxr-xr-x. 3 root     root  4096 11\u6708  1 01:05 logs\n-rw-r--r--. 1 wuqi2020 ftp  14978 7\u6708  19 2018 NOTICE.txt\n-rw-r--r--. 1 wuqi2020 ftp   1366 7\u6708  19 2018 README.txt\ndrwxr-xr-x. 2 wuqi2020 ftp   4096 7\u6708  19 2018 sbin\ndrwxr-xr-x. 4 wuqi2020 ftp     31 7\u6708  19 2018 share\n<\/code><\/pre>\n<p>\u901a\u8fc7\u4e0a\u9762\u5c55\u793a\u7684\u5b89\u88c5\u76ee\u5f55\u4e0b\u7684\u6587\u4ef6\u5939\uff0c\u5728\u540e\u9762\u7684\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\uff0c\u6211\u4eec\u9700\u8981\u5173\u6ce8\u4e0b\u9762\u8fd9\u56db\u4e2a\u6587\u4ef6\u5939\uff1a<\/p>\n<p>sbin\u4e0b\u5b58\u50a8\u7684\u662f\u81ea\u5e26\u7684\u811a\u672c\u60f3\u8981\u542f\u52a8\u8be5\u7ec4\u4ef6\u5c31\u5728\u8be5\u6587\u4ef6\u4e0b\u6267\u884c<\/p>\n<ul>\n<li>1.1-etc\u6587\u4ef6\u5939\u4e0b\u7684\u5185\u5bb9(\u5b58\u50a8\u914d\u7f6e\u6587\u4ef6\uff0c\u53c2\u6570\u8c03\u4f18\u8981\u5728\u8be5\u6587\u4ef6\u5939\u4e0b\u8fdb\u884c)<\/li>\n<\/ul>\n<p>\u8fd9\u4e9b\u5c5e\u6027\u90fd\u662f\u540e\u671f\u7ecf\u5e38\u8981\u64cd\u4f5c\u7684<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq2 etc]# cd hadoop\/\n[root@wq2 hadoop]# ll\n\u603b\u7528\u91cf 152\n-rw-r--r--  1 wuqi2020 ftp  1323 11\u6708  1 01:00 core-site.xml\n-rw-r--r--. 1 wuqi2020 ftp  4222 8\u6708  31 20:10 hadoop-env.sh\n-rw-r--r--  1 wuqi2020 ftp  1069 10\u6708 31 18:10 hdfs-site.xml\n-rw-r--r--. 1 wuqi2020 ftp 11801 7\u6708  19 2018 log4j.properties\n-rw-r--r--. 1 wuqi2020 ftp  1383 7\u6708  19 2018 mapred-env.sh\n-rw-r--r--  1 wuqi2020 ftp  1160 11\u6708  4 11:46 mapred-site.xml\n-rw-r--r--. 1 wuqi2020 ftp    12 9\u6708   9 01:20 slaves\n-rw-r--r--. 1 wuqi2020 ftp  4567 7\u6708  19 2018 yarn-env.sh\n-rw-r--r--  1 wuqi2020 ftp  1527 10\u6708 31 20:42 yarn-site.xml\n<\/code><\/pre>\n<ul>\n<li>1.2-data\u76ee\u5f55\u4e0b\u5b58\u50a8namenode\uff0cDataNode,secondarynamenode\u76f8\u5173\u6570\u636e\uff1b<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">drwxr-xr-x. 5 root root 51 4\u6708   5 22:55 dfs\n[root@wq1 data]# pwd\n\/opt\/hadoop-2.7.7\/data\n[root@wq1 data]# cd dfs\/\n[root@wq1 dfs]# ll\n\u603b\u7528\u91cf 0\ndrwx------. 3 root root 40 4\u6708   6 12:04 data\ndrwxr-xr-x. 3 root root 40 4\u6708   6 12:02 name\ndrwxr-xr-x. 3 root root 40 4\u6708   6 12:06 namesecondary\n<\/code><\/pre>\n<p>1.2.1-data\u76ee\u5f55\u4e0b<\/p>\n<pre><code class=\"language-bash line-numbers\">#\u5b58\u50a8datanode\u4fe1\u606f\u7684\n[root@wq2 subdir0]# pwd\n\/opt\/hadoop-2.7.7\/data\/dfs\/data\/current\/BP-1063901513-192.168.236.236-1604118984203\/current\/finalized\/subdir0\/subdir0\n[root@wq2 subdir0]# ll\n\u603b\u7528\u91cf 153492\n-rw-r--r-- 1 root root       54 11\u6708  1 01:14 blk_1073741826\n-rw-r--r-- 1 root root       11 11\u6708  1 01:14 blk_1073741826_1001.meta\n[root@wq2 subdir0]# cat blk_1073741826\n82      IPDDpvnbs\n94      QDTVDbatyc\n11      Ubk\n<\/code><\/pre>\n<ul>\n<li>1.2.2\u67e5\u770bname\u76ee\u5f55\u4e0b\u90fd\u6709\u54ea\u4e9b\u6587\u4ef6,\u8fd9\u4e5f\u662f\u6267\u884c\u683c\u5f0f\u5316\u547d\u4ee4\u540e\u8be5\u521d\u59cb\u5316\u7684\u5c31\u662f\u8be5\u76ee\u5f55\u4e0b\u7684\u6587\u4ef6<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">-rw-r--r--. 1 root root      42 4\u6708   7 10:58 edits_0000000000000000552-0000000000000000553\n-rw-r--r--. 1 root root      42 4\u6708   7 11:00 edits_0000000000000000554-0000000000000000555\n-rw-r--r--. 1 root root 1048576 4\u6708   7 11:00 edits_inprogress_0000000000000000556\n-rw-r--r--. 1 root root    5025 4\u6708   7 10:01 fsimage_0000000000000000551\n-rw-r--r--. 1 root root      62 4\u6708   7 10:01 fsimage_0000000000000000551.md5\n-rw-r--r--. 1 root root    5025 4\u6708   7 11:01 fsimage_0000000000000000555\n-rw-r--r--. 1 root root      62 4\u6708   7 11:01 fsimage_0000000000000000555.md5\n-rw-r--r--. 1 root root       4 4\u6708   7 11:00 seen_txid\n-rw-r--r--. 1 root root     207 4\u6708   6 11:49 VERSION\n[root@wq1 current]# cat seen_txid \n556\n[root@wq1 current]# cat fsimage_0000000000000000555.md5\n4b802aa7b396ee71f7ae2b7d4e317fa2 *fsimage_0000000000000000555\n<\/code><\/pre>\n<p>cat VERSION<\/p>\n<pre><code class=\"language-bash line-numbers\">#Sat Oct 31 18:11:37 CST 2020\nnamespaceID=356126353\nclusterID=CID-45dbc059-efc3-4323-a148-6860e4b4efc1\ncTime=0\nstorageType=NAME_NODE\nblockpoolID=BP-1042749061-192.168.236.237-1604139097617\nlayoutVersion=-63\n<\/code><\/pre>\n<ul>\n<li>1.3-logs\u76ee\u5f55\u4e0b<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">-rw-r--r--. 1 root root      715 4\u6708   4 11:44 hadoop-root-datanode-wq1.out.5\n-rw-r--r--. 1 root root  7080553 4\u6708   7 11:26 hadoop-root-namenode-wq1.log\n-rw-r--r--. 1 root root     5007 4\u6708   6 12:19 hadoop-root-namenode-wq1.out\n<\/code><\/pre>\n<ul>\n<li>1.4 sbin\u76ee\u5f55\u4e0b(\u81ea\u5e26shell\u547d\u4ee4)<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">-rwxr-xr-x. 1 wuqi2020 ftp 2752 7\u6708  19 2018 distribute-exclude.sh\n-rwxr-xr-x. 1 wuqi2020 ftp 6452 7\u6708  19 2018 hadoop-daemon.sh\n-rwxr-xr-x. 1 wuqi2020 ftp 1360 7\u6708  19 2018 hadoop-daemons.sh\n<\/code><\/pre>\n<p><strong>Fsimage\u6587\u4ef6<\/strong>\uff1aHDFS\u6587\u4ef6\u7cfb\u7edf\u5143\u6570\u636e\u7684\u4e00\u4e2a\u6c38\u4e45\u6027\u7684\u68c0\u67e5\u70b9\uff0c\u5176\u4e2d\u5305\u542bHDFS\u6587\u4ef6\u7cfb\u7edf\u7684\u6240\u6709\u76ee\u5f55\u548c\u6587\u4ef6idnode\u7684\u5e8f\u5217\u5316\u4fe1\u606f\u3002 FSImage\u6587\u4ef6\u4e5f\u6709\u4e24\u79cd\u72b6\u6001, finalized\u548ccheckpoint\uff0c finalized\u8868\u793a\u5df2\u7ecf\u6301\u4e45\u5316\u78c1\u76d8\u7684\u6587\u4ef6\uff0c\u6587\u4ef6\u540d\u5f62\u5f0f: fsimage_[end-txid], checkpoint\u8868\u793a\u5408\u5e76\u4e2d\u7684fsimage, 2.x\u7248\u672ccheckpoint\u8fc7\u7a0b\u5728Standby Namenode(SNN)\u4e0a\u8fdb\u884c\uff1b<\/p>\n<ul>\n<li>\u67e5\u770bFsimage\u6587\u4ef6<\/li>\n<\/ul>\n<pre><code class=\"language-properties line-numbers\">hdfs oiv -p Delimited -i fsimage_0000000000000000551 fsimage_0000000000000000551.xml\n<\/code><\/pre>\n<p>\u5f62\u5f0f\u5982\u4e0b(\u6587\u4ef6\u6570)<\/p>\n<pre><code class=\"language-bash line-numbers\">\/tmp\/hadoop-yarn\/staging\/root\/.staging\/job_1617615663836_0025\/COMMIT_FAIL       1       2021-04-07 00:17        2021-04-07 00:17        134217728   rw-r--r--        root    supergroup\n\/tmp\/hadoop-yarn\/staging\/history\/done\/2021\/04\/07\/000000\/job_1617615663836_0025-1617725348829-root-insert+into+table+test_bu...test_bucket_tmp1%28Stage-1617725877707-1-4-KILLED-default-1617725358318.jhist  1       2021-04-07 00:17        2021-04-07 00:17        134217728       1       46364   0   rwxrwx---        root    supergroup\n\/tmp\/hadoop-yarn\/staging\/history\/done\/2021\/04\/07\/000000\/job_1617615663836_0025_conf.xml 1       2021-04-07 00:17        2021-04-07 00:17        134217728    1       242647  0       0       rwxrwx---       root    supergroup\n\/tmp\/logs\/root\/logs\/application_1617615663836_0025\/wq1_33210    1       2021-04-07 00:30        2021-04-07 00:30        134217728       1       349379       0       0       rw-r-----       root    root\n21\/04\/12 11:25:30 INFO offlineImageViewer.PBImageTextWriter: Outputted 3 INodes.\n<\/code><\/pre>\n<p><strong>Edits\u6587\u4ef6<\/strong>\uff1a\u5b58\u653eHDFS\u6587\u4ef6\u7cfb\u7edf\u7684\u6240\u6709\u66f4\u65b0\u64cd\u4f5c\u7684\u8def\u5f84\uff0c\u6587\u4ef6\u7cfb\u7edf\u5ba2\u6237\u7aef\u6267\u884c\u7684\u6240\u6709\u5199\u64cd\u4f5c\u9996\u5148\u4f1a\u88ab\u8bb0\u5f55\u5230edits\u6587\u4ef6\u4e2d\u3002<code>EditLog<\/code>\u6587\u4ef6\u6709\u4e24\u79cd\u72b6\u6001\uff1a inprocess\u548cfinalized, inprocess\u8868\u793a\u6b63\u5728\u5199\u7684\u65e5\u5fd7\u6587\u4ef6\uff0c\u6587\u4ef6\u540d\u5f62\u5f0f:editsinprocess[start-txid]\uff0cfinalized\u8868\u793a\u5df2\u7ecf\u5199\u5b8c\u7684\u65e5\u5fd7\u6587\u4ef6,\u6587\u4ef6\u540d\u5f62\u5f0f\uff1aedits[start-txid][end-txid]<br \/>\n&#8211; \u67e5\u770bedits\u6587\u4ef6<\/p>\n<pre><code class=\"language-properties line-numbers\">#\u5c06edits\u6587\u4ef6\u8f6c\u4e3axml\u6587\u4ef6\u65b9\u4fbf\u67e5\u770b\nhdfs oev -i edits_inprogress_0000000000000014085 -o edits_inprogress_0000000000000014085.xml\n#\u8f6c\u56de\u539f\u672c\u7684\u4e8c\u8fdb\u5236\u6587\u4ef6\nhdfs oev -i edits_inprogress_0000000000000014085.xml -o edits_inprogress_0000000000000014085 -p binary\n<\/code><\/pre>\n<p>edits\u6587\u4ef6\u7ecf\u8fc7hdfs\u81ea\u5e26\u547d\u4ee4\u8f6c\u6362\u540e\u957f\u7740\u4e0b\u9762\u8fd9\u4e2a\u6837\u5b50<\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">&lt;?xml version=\"1.0\" encoding=\"UTF-8\"?&gt;\n&lt;EDITS&gt;\n  &lt;EDITS_VERSION&gt;-63&lt;\/EDITS_VERSION&gt;\n&lt;RECORD&gt;\n    &lt;OPCODE&gt;OP_RENAME_OLD&lt;\/OPCODE&gt;\n    &lt;DATA&gt;\n      &lt;TXID&gt;14084&lt;\/TXID&gt;\n      &lt;LENGTH&gt;0&lt;\/LENGTH&gt;\n      &lt;SRC&gt;\/logs\/MapReduceOnHue-1.0-SNAPSHOT.jar.tmp&lt;\/SRC&gt;\n      &lt;DST&gt;\/logs\/MapReduceOnHue-1.0-SNAPSHOT.jar&lt;\/DST&gt;\n      &lt;TIMESTAMP&gt;1617631563000&lt;\/TIMESTAMP&gt;\n      &lt;RPC_CLIENTID&gt;&lt;\/RPC_CLIENTID&gt;\n      &lt;RPC_CALLID&gt;-2&lt;\/RPC_CALLID&gt;\n    &lt;\/DATA&gt;\n  &lt;\/RECORD&gt;\n&lt;\/EDITS&gt;\n<\/code><\/pre>\n<p><strong>seen_txid<\/strong>\u6587\u4ef6\u4fdd\u5b58\u7684\u662f\u4e00\u4e2a\u6570\u5b57\uff0c\u5c31\u662f\u6700\u540e\u4e00\u4e2aedits_\u7684\u6570\u5b57\u6bcf\u6b21Namenode\u542f\u52a8\u7684\u65f6\u5019\u90fd\u4f1a\u5c06fsimage\u6587\u4ef6\u8bfb\u5165\u5185\u5b58\uff0c\u5e76\u4ece00001\u5f00\u59cb\u5230seen_txid\u4e2d\u8bb0\u5f55\u7684\u6570\u5b57\u4f9d\u6b21\u6267\u884c\u6bcf\u4e2aedits\u91cc\u9762\u7684\u66f4\u65b0\u64cd\u4f5c\uff0c\u4fdd\u8bc1\u5185\u5b58\u4e2d\u7684\u5143\u6570\u636e\u4fe1\u606f\u662f\u6700\u65b0\u7684\u3001\u540c\u6b65\u7684\uff0c\u53ef\u4ee5\u770b\u6210Namenode\u542f\u52a8\u7684\u65f6\u5019\u5c31\u5c06fsimage\u548cedits\u6587\u4ef6\u8fdb\u884c\u4e86\u5408\u5e76\u3002seen_txid,\u4fdd\u5b58\u7684\u662f\u4e00\u4e2a\u4e8b\u52a1ID\uff0c\u8fd9\u4e2a\u4e8b\u52a1ID\u662fEditLog\u6700\u65b0\u7684\u4e00\u4e2a\u7ed3\u675f\u4e8b\u52a1id\uff0c\u5f53NameNode\u91cd\u542f\u65f6\uff0c\u4f1a\u987a\u5e8f\u904d\u5386\u4eceedits_0000000000000000001\u5230seen_txid\u6240\u8bb0\u5f55\u7684txid\u6240\u5728\u7684\u65e5\u5fd7\u6587\u4ef6\uff0c\u8fdb\u884c\u5143\u6570\u636e\u6062\u590d\uff0c\u5982\u679c\u8be5\u6587\u4ef6\u4e22\u5931\u6216\u8bb0\u5f55\u7684\u4e8b\u52a1ID\u6709\u95ee\u9898\uff0c\u4f1a\u9020\u6210\u6570\u636e\u5757\u4fe1\u606f\u7684\u4e22\u5931\u3002<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 current]# cat seen_txid \n6283\n<\/code><\/pre>\n<hr \/>\n<p><strong>namenode:<\/strong> \u7b2c\u4e00\u6b21\u542f\u52a8namenode\u683c\u5f0f\u5316\u540e\uff0c\u521b\u5efafsimage\u548cedits\u6587\u4ef6\u3002\u5982\u679c\u4e0d\u662f\u7b2c\u4e00\u6b21\u542f\u52a8\uff0c\u76f4\u63a5\u52a0\u8f7d\u7f16\u8f91\u65e5\u5fd7\u548c\u955c\u50cf\u6587\u4ef6\u5230\u5185\u5b58\u3002\u5ba2\u6237\u7aef\u5bf9\u5143\u6570\u636e\u8fdb\u884c\u589e\u5220\u6539\u7684\u8bf7\u6c42namenode\u8bb0\u5f55\u64cd\u4f5c\u65e5\u5fd7\uff0c\u66f4\u65b0\u6eda\u52a8\u65e5\u5fd7\u3002namenode\u5728\u5185\u5b58\u4e2d\u5bf9\u6570\u636e\u8fdb\u884c\u589e\u5220\u6539\u3002namenode\u5728\u52a0\u8f7dfsimage\u8fc7\u7a0b\u5176\u5b9e\u2fae\u5e38\u7b80\u5355\uff0c\u5c31\u662f\u4ecefsimage\u4e2d\u4e0d\u505c\u7684\u987a\u5e8f\u8bfb\u53d6\u2f42\u4ef6\u548c\u2f6c\u5f55\u7684\u5143\u6570\u636e\u4fe1\u606f\uff0c\u5e76\u5728\u5185\u5b58\u4e2d\u6784\u5efa\u6574\u4e2anamespace\uff0c\u540c\u65f6\u5c06\u6bcf\u4e2a\u2f42\u4ef6\u5bf9\u5e94\u7684blockid\u4fdd\u5b58\u2f0aBlocksMap\u4e2d\uff0c\u6b64\u65f6BlocksMap\u4e2d\u6bcf\u4e2ablock\u5bf9\u5e94\u7684datanodes\u5217\u8868\u6682\u65f6\u4e3a\u7a7a\u3002\u5f53fsimage\u52a0\u8f7d\u5b8c\u6bd5\u540e\uff0c\u6574\u4e2aHDFS\u7684\u2f6c\u5f55\u7ed3\u6784\u5728\u5185\u5b58\u4e2d\u5c31\u5df2\u7ecf\u521d\u59cb\u5316\u5b8c\u6bd5\uff0c\u6240\u7f3a\u7684\u5c31\u662f\u6bcf\u4e2a\u2f42\u4ef6\u5bf9\u5e94\u7684block\u5bf9\u5e94\u7684datanode\u5217\u8868\u4fe1\u606f\u3002\u8fd9\u4e9b\u4fe1\u606f\u9700\u8981\u4ecedatanode\u7684blockReport\u4e2d\u83b7\u53d6\uff0c\u6240\u4ee5\u52a0\u8f7dfsimage\u5b8c\u6bd5\u540e\uff0cnamenode\u8fdb\u7a0b\u8fdb\u2f0arpc\u7b49\u5f85\u72b6\u6001\uff0c\u7b49\u5f85\u6240\u6709\u7684datanodes\u53d1\u9001blockRep<br \/>\n<strong>Secondary NameNode<\/strong>:\u8be2\u95eenamenode\u662f\u5426\u9700\u8981checkpoint\u3002\u76f4\u63a5\u5e26\u56denamenode\u662f\u5426\u68c0\u67e5\u7ed3\u679c\u3002namenode\u6eda\u52a8\u6b63\u5728\u5199\u7684edits\u65e5\u5fd7\u5c06\u6eda\u52a8\u524d\u7684\u7f16\u8f91\u65e5\u5fd7\u548c\u955c\u50cf\u6587\u4ef6\u62f7\u8d1d\u5230Secondary NameNodeSecondary NameNode\u52a0\u8f7d\u7f16\u8f91\u65e5\u5fd7\u548c\u955c\u50cf\u6587\u4ef6\u5230\u5185\u5b58\uff0c\u5e76\u5408\u5e76\u3002\u751f\u6210\u65b0\u7684\u955c\u50cf\u6587\u4ef6fsimage.chkpoint\u62f7\u8d1dfsimage.chkpoint\u5230namenode\u5c06fsimage.chkpoint\u91cd\u65b0\u547d\u540d\u6210fsimage<\/p>\n<p><strong>datanode<\/strong>:\u4e00\u4e2a\u6570\u636e\u5757\u5728datanode\u4e0a\u4ee5\u6587\u4ef6\u5f62\u5f0f\u5b58\u50a8\u5728\u78c1\u76d8\u4e0a\uff0c\u5305\u62ec\u4e24\u4e2a\u6587\u4ef6\uff0c\u4e00\u4e2a\u662f\u6570\u636e\u672c\u8eab\uff0c\u4e00\u4e2a\u662f\u5143\u6570\u636e\u5305\u62ec\u6570\u636e\u5757\u7684\u957f\u5ea6\uff0c\u5757\u6570\u636e\u7684\u6821\u9a8c\u548c\uff0c\u4ee5\u53ca\u65f6\u95f4\u6233\u3002DataNode\u542f\u52a8\u540e\u5411namenode\u6ce8\u518c\uff0c\u901a\u8fc7\u540e\uff0c\u5468\u671f\u6027\uff081\u5c0f\u65f6\uff09\u7684\u5411namenode\u4e0a\u62a5\u6240\u6709\u7684\u5757\u4fe1\u606f\u3002<\/p>\n<blockquote><p>\n  checkpoint\n<\/p><\/blockquote>\n<ul>\n<li>\u96c6\u7fa4\u542f\u52a8\u65f6\uff0c\u52a0\u8f7dfsimage\u955c\u50cf\u6587\u4ef6\u5230\u5185\u5b58\uff0c\u5982\u679c\u662f\u7b2c\u4e00\u542f\u52a8\u96c6\u7fa4\u6216\u8005\u96c6\u7fa4\u6b63\u5e38\u5173\u95ed\u4e4b\u540e\u91cd\u542f\u6b64\u65f6nameNode\u4f1a\u5728\u786c\u76d8\u4e2d\u5408\u5e76\u4e00\u4e2afsimage\u955c\u50cf<\/li>\n<li>seconddaryNameNode\u5b9a\u65f6\uff081\u5206\u949f\uff09\u53d1\u9001\u68c0\u9a8c\u7ed9nameNode\uff0c\u67e5\u770b\u662f\u5426\u9700\u8981\u8fdb\u884c\u5408\u5e76<\/li>\n<li>\u5f97\u77e5nameNode\u9700\u8981\u8fdb\u884c\u5143\u6570\u636e\u5408\u5e76<\/li>\n<li>seconddaryNameNode\u5411nameNode\u53d1\u9001\u5408\u5e76\u8bf7\u6c42<\/li>\n<li>nameNode\u5c06edits_inprogress_000095 \u6839\u636eseen_txid\u8fdb\u884c\u56de\u6eda\uff0c\u5e76\u4e14\u751f\u6210\u4e00\u4e2a\u65b0\u7684\u7a7a\u7684edits_inprogress_000096\uff0c\u7ee7\u7eed\u8bb0\u5f55\u64cd\u4f5c\u65e5\u5fd7<\/li>\n<li>secondaryNameNode\u5c06\u56de\u6eda\u7684edits\u548c\u6700\u65b0\u7684fsiamge\u8fdb\u884c\u672c\u5730\u62c9\u53d6<\/li>\n<li>secondaryNameNode\u5c06edits\u548c\u6700\u65b0\u7684fsiamge\u8fdb\u884c\u5408\u5e76\uff0c\u5728\u5185\u5b58\u4e2d\u6839\u636eedits\u4fee\u6539fsiamge<\/li>\n<li>secondaryNameNode\u5c06\u5408\u5e76\u540e\u7684fsiamge\u63a8\u9001\u56denamenode\u3002\u5e76\u5728\u672c\u5730\u4fdd\u5b58\u4e00\u4efd\u3002<\/li>\n<\/ul>\n<p>\u89e6\u53d1\u914d\u7f6e<\/p>\n<pre><code class=\"language-bash line-numbers\">dfs.namenode.checkpoint.check.period=60 ##\u68c0\u67e5\u89e6\u53d1\u6761\u4ef6\u662f\u5426\u6ee1\u8db3\u7684\u9891\u7387\uff0c60 \u79d2\ndfs.namenode.checkpoint.dir=file:\/\/<span class=\"katex math inline\">{hadoop.tmp.dir}\/dfs\/namesecondary\n##\u4ee5\u4e0a\u4e24\u4e2a\u53c2\u6570\u505a checkpoint \u64cd\u4f5c\u65f6\uff0csecondary namenode \u7684\u672c\u5730\u5de5\u4f5c\u76ee\u5f55\ndfs.namenode.checkpoint.edits.dir=<\/span>{dfs.namenode.checkpoint.dir}\ndfs.namenode.checkpoint.max-retries=3 ##\u6700\u5927\u91cd\u8bd5\u6b21\u6570\ndfs.namenode.checkpoint.period=3600 ##\u4e24\u6b21 checkpoint \u4e4b\u95f4\u7684\u65f6\u95f4\u95f4\u9694 3600 \u79d2\ndfs.namenode.checkpoint.txns=1000000 ##\u4e24\u6b21 checkpoint \u4e4b\u95f4\u6700\u5927\u7684\u64cd\u4f5c\u8bb0\u5f55  \n<\/code><\/pre>\n<p>Namenode \u548c SecondaryNamenode \u7684\u5de5\u4f5c\u76ee\u5f55\u5b58\u50a8\u7ed3\u6784\u5b8c\u5168\u76f8\u540c\uff0c\u6240\u4ee5\uff0c\u5f53 Namenode \u6545\u969c<br \/>\n\u9000\u51fa\u9700\u8981\u91cd\u65b0\u6062\u590d\u65f6\uff0c\u53ef\u4ee5\u4eceSecondaryNamenode\u7684\u5de5\u4f5c\u76ee\u5f55\u4e2d\u5c06fsimage\u62f7\u8d1d\u5230Namenode<br \/>\n\u7684\u5de5\u4f5c\u76ee\u5f55\uff0c\u4ee5\u6062\u590d namenode \u7684\u5143\u6570\u636e.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"%E5%90%91hdfs%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6%E4%B8%8B%E8%BD%BD%E6%96%87%E4%BB%B6\"><\/span>\u5411hdfs\u4e0a\u4f20\u6587\u4ef6\u4e0b\u8f7d\u6587\u4ef6<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>\u5411hdfs\u4e0a\u4f20\u6587\u4ef6\u6709\u4e09\u79cd\u9014\u5f84\uff1a\u4f7f\u7528Java\u4ee3\u7801\u4f20\u6570\u636e\uff0c\u4f7f\u7528hdfs\u7ec4\u4ef6\u81ea\u8eab\u7684\u547d\u4ee4\u4e0a\u4f20\u6587\u4ef6\uff0c\u4f7f\u7528\u5176\u4ed6\u7ec4\u5c06\u5411hdfs\u4e0a\u4f20\u6587\u4ef6<\/p>\n<h3><span class=\"ez-toc-section\" id=\"%E4%BD%BF%E7%94%A8Java%E4%BB%A3%E7%A0%81%E5%90%91hdfs%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\"><\/span>\u4f7f\u7528Java\u4ee3\u7801\u5411hdfs\u4e0a\u4f20\u6587\u4ef6<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre><code class=\"language-java line-numbers\">    public static void main(String[] args) throws Exception{\n        boolean flag;\n        \/\/ HDFS \u7684\u64cd\u4f5c\n        \/\/ HDFS \u7684API\u5165\u53e3\u7c7b\u662fFileSystem\n        \/\/ FileSystem \u6709\u591a\u4e2a\u5b9e\u73b0\u7c7b\n        \/\/ FileSystem \u53ef\u4ee5\u6839\u636eURI\u517c\u5bb9\u5404\u79cd\u5404\u6837\u4e0d\u540c\u7684\u6587\u4ef6\u7cfb\u7edf\n        \/\/ 0. \u521b\u5efa\u914d\u7f6e\u6587\u4ef6\u5bf9\u8c61\n        \/\/ \u8fd9\u4e2a\u5bf9\u8c61\u53ef\u4ee5\u5728\u7a0b\u5e8f\u8fd0\u884c\u65f6\u4e34\u65f6\u4fee\u6539Hadoop\u7684\u4e00\u4e9b\u914d\u7f6e\u53c2\u6570\n        \/\/ \u4f18\u5148\u7ea7  \u4ee3\u7801\u4e2d  &gt; resource\u7684\u914d\u7f6e\u6587\u4ef6  &gt; \u96c6\u7fa4\n        Configuration conf = new Configuration();\n        conf.set(\"dfs.replication\", \"1\");\n        \/\/ 1. \u901a\u8fc7FileSystem\u521b\u5efa  HDFS\u6587\u4ef6\u7cfb\u7edf\u5bf9\u8c61\n        FileSystem fs = FileSystem.get(\n                URI.create(\"hdfs:\/\/192.168.236.236:9000\/\"),\n                conf,\n                \"root\"\n        );\n        upload(fs);\n        download\uff08fs\uff09\n    }\n\/\/\u4e0a\u4f20\u6587\u4ef6 (\u8986\u76d6)\nprivate static void upload(FileSystem fs) throws IOException {\n        fs.copyFromLocalFile(\n                new Path(\"C:\\\\Users\\\\Administrator\\\\Desktop\\\\access.log\"),\n                new Path(\"\/java-hdfs\/\")\n        );\n    }\n\/\/\u4e0b\u8f7d\u6587\u4ef6\n private static void download(FileSystem fs) throws IOException {\n        fs.copyToLocalFile(\n                new Path(\"\/up1\/log.txt\"),\n                new Path(\"C:\\\\Users\\\\Administrator\\\\Desktop\\\\access.log\")\n        );\n    }\n<\/code><\/pre>\n<h3><span class=\"ez-toc-section\" id=\"%E4%BD%BF%E7%94%A8hdfs%E5%91%BD%E4%BB%A4%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\"><\/span>\u4f7f\u7528hdfs\u547d\u4ee4\u4e0a\u4f20\u6587\u4ef6<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li>\u628ahdfs\u4e0a\u7684\u591a\u4e2a\u76ee\u5f55\u4e0b\u7684\u6587\u4ef6\u5408\u5e76\u4e3a\u4e00\u4e2a\u6587\u4ef6<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">hdfs dfs -cat files | hdfs dfs -copyFromLocal - newfile\n<\/code><\/pre>\n<ul>\n<li>\u5347\u526f\u672c\u6570<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">hdfs dfs -setrep -R -w 3 hdfs:\/\/192.168.236.236:50070\/\/hdfs-hive\n<\/code><\/pre>\n<ul>\n<li>\u964d\u526f\u672c\u6570<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">hdfs dfs -setrep -R -w 2 hdfs:\/\/192.168.236.236:50070\/\/\/hdfs-hive\n<\/code><\/pre>\n<ul>\n<li>\u4e0a\u4f20\u6587\u4ef6<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">hdfs dfs -put \/root\/oldfile \/newfile\n<\/code><\/pre>\n<ul>\n<li>\u5220\u9664\u6587\u4ef6\/\u6587\u4ef6\u5939<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">hdfs dfs -rm  \/user\/hive\/warehouse\/\u6587\u4ef6\u7684\u540d\nhdfs dfs -rm -r \/user\/hive\/warehouse\/\u6587\u4ef6\u5939\u7684\u540d\n<\/code><\/pre>\n<ul>\n<li>\u6c47\u603b<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 etc]# hdfs dfs -help\n-help\uff1a\u8f93\u51fa\u8fd9\u4e2a\u547d\u4ee4\u53c2\u6570\n-ls: \u663e\u793a\u76ee\u5f55\u4fe1\u606f\n-mkdir\uff1a\u5728hdfs\u4e0a\u521b\u5efa\u76ee\u5f55\n-moveFromLocal\u4ece\u672c\u5730\u526a\u5207\u7c98\u8d34\u5230hdfs\n-appendToFile  \uff1a\u8ffd\u52a0\u4e00\u4e2a\u6587\u4ef6\u5230\u5df2\u7ecf\u5b58\u5728\u7684\u6587\u4ef6\u672b\u5c3e\n-cat \uff1a\u663e\u793a\u6587\u4ef6\u5185\u5bb9\n-tail\uff1a\u663e\u793a\u4e00\u4e2a\u6587\u4ef6\u7684\u672b\u5c3e\n-chgrp \u3001-chmod\u3001-chown\uff1alinux\u6587\u4ef6\u7cfb\u7edf\u4e2d\u7684\u7528\u6cd5\u4e00\u6837\uff0c\u4fee\u6539\u6587\u4ef6\u6240\u5c5e\u6743\u9650\n-copyFromLocal\uff1a\u4ece\u672c\u5730\u6587\u4ef6\u7cfb\u7edf\u4e2d\u62f7\u8d1d\u6587\u4ef6\u5230hdfs\u8def\u5f84\u53bb\n-copyToLocal\uff1a\u4ecehdfs\u62f7\u8d1d\u5230\u672c\u5730\n-cp \uff1a\u4ecehdfs\u7684\u4e00\u4e2a\u8def\u5f84\u62f7\u8d1d\u5230hdfs\u7684\u53e6\u4e00\u4e2a\u8def\u5f84\n-mv\uff1a\u5728hdfs\u76ee\u5f55\u4e2d\u79fb\u52a8\u6587\u4ef6\n-get\uff1a\u7b49\u540c\u4e8ecopyToLocal\uff0c\u5c31\u662f\u4ecehdfs\u4e0b\u8f7d\u6587\u4ef6\u5230\u672c\u5730\n-getmerge  \uff1a\u5408\u5e76\u4e0b\u8f7d\u591a\u4e2a\u6587\u4ef6\uff0c\u6bd4\u5982hdfs\u7684\u76ee\u5f55 \/hj\/\u4e0b\u6709\u591a\u4e2a\u6587\u4ef6:log.1, log.2,log.3,...\n-put\uff1a\u7b49\u540c\u4e8ecopyFromLocal\n-rm\uff1a\u5220\u9664\u6587\u4ef6\u6216\u6587\u4ef6\u5939\n-rmdir\uff1a\u5220\u9664\u7a7a\u76ee\u5f55\n-df \uff1a\u7edf\u8ba1\u6587\u4ef6\u7cfb\u7edf\u7684\u53ef\u7528\u7a7a\u95f4\u4fe1\u606f\n-du\u7edf\u8ba1\u6587\u4ef6\u5939\u7684\u5927\u5c0f\u4fe1\u606f\n-setrep\uff1a\u8bbe\u7f6ehdfs\u4e2d\u6587\u4ef6\u7684\u526f\u672c\u6570\u91cf\n<\/code><\/pre>\n<h3><span class=\"ez-toc-section\" id=\"%E4%BD%BF%E7%94%A8%E5%85%B6%E4%BB%96%E7%BB%84%E4%BB%B6\"><\/span>\u4f7f\u7528\u5176\u4ed6\u7ec4\u4ef6<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>\u5176\u4ed6\u7ec4\u4ef6\u4e0d\u6b62\u6211\u4e0b\u9762\u5199\u7684flume\uff0chive\uff0csqoop<\/p>\n<h4><span class=\"ez-toc-section\" id=\"%E4%BD%BF%E7%94%A8sqoop%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\"><\/span>\u4f7f\u7528sqoop\u4e0a\u4f20\u6587\u4ef6<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<ol>\n<li>\u4e0a\u4f20\u5168\u91cf\u6570\u636e<\/li>\n<\/ol>\n<pre><code class=\"language-bash line-numbers\">sqoop import \\\n --connect jdbc:mysql:\/\/192.168.xxx.xxx:3316\/testdb \\\n --username root \\\n --password 123456 \\\n --query \u201cselect * from test_table where \\$CONDITIONS\u201d \\\n --target-dir \/user\/root\/person_all \\ \n --fields-terminated-by \u201c,\u201d \\\n --hive-drop-import-delims \\\n --null-string \u201c\\\\N\u201d \\\n --null-non-string \u201c\\\\N\u201d \\\n --split-by id \\\n -m 6 \\\n<\/code><\/pre>\n<ol start=\"2\">\n<li>\u4e0a\u4f20\u589e\u91cf\u6570\u636e<\/li>\n<\/ol>\n<p>\u6b64\u65b9\u5f0f\u8981\u6c42\u539f\u6709\u8868\u4e2d\u6709time\u5b57\u6bb5\uff0c\u5b83\u80fd\u6307\u5b9a\u4e00\u4e2a\u65f6\u95f4\u6233\uff0c\u8ba9Sqoop\u628a\u8be5\u65f6\u95f4\u6233\u4e4b\u540e\u7684\u6570\u636e\u5bfc\u5165\u81f3Hadoop\uff08\u8fd9\u91cc\u4e3aHDFS\uff09\u3002\u56e0\u4e3a\u540e\u7eed\u8ba2\u5355\u53ef\u80fd\u72b6\u6001\u4f1a\u53d8\u5316\uff0c\u53d8\u5316\u540etime\u5b57\u6bb5\u65f6\u95f4\u6233\u4e5f\u4f1a\u53d8\u5316\uff0c\u6b64\u65f6Sqoop\u4f9d\u7136\u4f1a\u5c06\u76f8\u540c\u72b6\u6001\u66f4\u6539\u540e\u7684\u8ba2\u5355\u5bfc\u5165HDFS\uff0c\u5f53\u7136\u6211\u4eec\u53ef\u4ee5\u6307\u5b9amerge-key\u53c2\u6570\u4e3aorser_id\uff0c\u8868\u793a\u5c06\u540e\u7eed\u65b0\u7684\u8bb0\u5f55\u4e0e\u539f\u6709\u8bb0\u5f55\u5408\u5e76<\/p>\n<pre><code class=\"language-bash line-numbers\"># \u5c06\u65f6\u95f4\u5217\u5927\u4e8e\u7b49\u4e8e\u9608\u503c\u7684\u6570\u636e\u589e\u91cf\u5bfc\u5165HDFS\n sqoop import \\\n   --connect jdbc:mysql:\/\/192.168.xxx.xxx:3316\/testdb \\\n   --username root \\\n   --password transwarp \\\n   --query \u201cselect order_id, name from order_table where \\$CONDITIONS\u201d \\\n   --target-dir \/user\/root\/order_all \\ \n   --split-by id \\\n   -m 4  \\\n   --incremental lastmodified \\\n   --merge-key order_id \\\n   --check-column time \\\n   # remember this date !!!\n   --last-value \u201c2014-11-09 21:00:00\u201d  \n<\/code><\/pre>\n<h4><span class=\"ez-toc-section\" id=\"%E4%BD%BF%E7%94%A8flume%E5%90%91hdfs%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6\"><\/span>\u4f7f\u7528flume\u5411hdfs\u4e0a\u4f20\u6587\u4ef6<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>\u4f7f\u7528flume\u5411hdfs\u4e0a\u4f20\u6587\u4ef6\u53ef\u5206\u4e3a\u4e24\u6b65\uff0c\u7b2c\u4e00\u6b65\u5199flume\u811a\u672c\uff08\u811a\u672c\u540d\u968f\u610f\uff0c\u540e\u7f00\u4e3a.conf\u5c31\u884c\uff09\uff0c\u7b2c\u4e8c\u6b65\u6267\u884c\u811a\u672c\u3002<br \/>\n\u8be5\u811a\u672c\u540d\u4e3aflume-hdfs.conf<\/p>\n<pre><code class=\"language-bash line-numbers\"># Name the components on this agent \na2.sources = r2\na2.sinks = k2 \na2.channels = c2\n\n# Describe\/configure the source \na2.sources.r2.type = exec\na2.sources.r2.command = tail -F \/opt\/hive\/logs\/hive.log\na2.sources.r2.shell = \/bin\/bash -c\n\n# Describe the sink \na2.sinks.k2.type = hdfs\na2.sinks.k2.hdfs.path = hdfs:\/\/test01:9000\/flume\/%Y%m%d\/%H\n#\u4e0a\u4f20\u6587\u4ef6\u7684\u524d\u7f00 \na2.sinks.k2.hdfs.filePrefix = logs-\n#\u662f\u5426\u6309\u7167\u65f6\u95f4\u6eda\u52a8\u6587\u4ef6\u5939 \na2.sinks.k2.hdfs.round = true\n#\u591a\u5c11\u65f6\u95f4\u5355\u4f4d\u521b\u5efa\u4e00\u4e2a\u65b0\u7684\u6587\u4ef6\u5939 \na2.sinks.k2.hdfs.roundValue = 1\n#\u91cd\u65b0\u5b9a\u4e49\u65f6\u95f4\u5355\u4f4d \na2.sinks.k2.hdfs.roundUnit = hour\n#\u662f\u5426\u4f7f\u7528\u672c\u5730\u65f6\u95f4\u6233 \na2.sinks.k2.hdfs.useLocalTimeStamp = true\n#\u79ef\u6512\u591a\u5c11\u4e2aEvent \u624d flush \u5230 HDFS \u4e00\u6b21\na2.sinks.k2.hdfs.batchSize = 1000\n#\u8bbe\u7f6e\u6587\u4ef6\u7c7b\u578b\uff0c\u53ef\u652f\u6301\u538b\u7f29 \na2.sinks.k2.hdfs.fileType = DataStream\n#\u591a\u4e45\u751f\u6210\u4e00\u4e2a\u65b0\u7684\u6587\u4ef6 \na2.sinks.k2.hdfs.rollInterval = 600\n#\u8bbe\u7f6e\u6bcf\u4e2a\u6587\u4ef6\u7684\u6eda\u52a8\u5927\u5c0f \na2.sinks.k2.hdfs.rollSize = 134217700\n#\u6587\u4ef6\u7684\u6eda\u52a8\u4e0e Event \u6570\u91cf\u65e0\u5173 \na2.sinks.k2.hdfs.rollCount = 0\n#\u6700\u5c0f\u5197\u4f59\u6570 \na2.sinks.k2.hdfs.minBlockReplicas = 1\n\n# Use a channel which buffers events in memory \na2.channels.c2.type = memory \na2.channels.c2.capacity = 1000\na2.channels.c2.transactionCapacity = 100\n\n# Bind the source and sink to the channel\na2.sources.r2.channels = c2 \na2.sinks.k2.channel = c2\n<\/code><\/pre>\n<p>\u5728flume\u7684bin\u4e0b\u6267\u884c\u8be5\u811a\u672c\uff08\u5982\u679c\u914d\u7f6e\u4e86\u73af\u5883\u53d8\u91cf\uff09<\/p>\n<pre><code class=\"language-bash line-numbers\">flume-ng agent\\\n--conf \/opt\/flume-1.9.0\/conf\/ \\\n--name a2 \\\n--conf-file \/root\/flume_conf\/flume-hdfs.conf \\\n-Dflume.root.logger=INFO,console\n<\/code><\/pre>\n<h4><span class=\"ez-toc-section\" id=\"%E4%BD%BF%E7%94%A8hive%E5%B0%86%E6%95%B0%E6%8D%AE%E5%AF%BC%E5%88%B0hdfs%E4%B8%8A\"><\/span>\u4f7f\u7528hive\u5c06\u6570\u636e\u5bfc\u5230hdfs\u4e0a<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<pre><code class=\"language-sql line-numbers\">insert overwrite directory '\/user\/hive\/testhive'\nselect * from test ;\n<\/code><\/pre>\n<h1><span class=\"ez-toc-section\" id=\"%E4%B8%89%E3%80%81hdfs%E6%A0%B8%E5%BF%83%E9%97%AE%E9%A2%98\"><\/span>\u4e09\u3001hdfs\u6838\u5fc3\u95ee\u9898<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<blockquote><p>\n  hdfs\u5355\u70b9\u6545\u969c\u548c\u5185\u5b58\u53d7\u9650\u95ee\u9898\n<\/p><\/blockquote>\n<p><code>HDFS HA<\/code>\uff1a\u901a\u8fc7\u4e3b\u5907NameNode\u89e3\u51b3 \u4e00\u4e2a\u96c6\u7fa4\u4e2d\u53ea\u80fd\u6709\u4e00\u4e2aNameNode\u5904\u4e8e\u5de5\u4f5c\u72b6\u6001 \u5f53\u4e3bNameNode\u53d1\u9001\u6545\u969c \u5219\u5207\u6362\u5230\u5907NameNode\u4e0a,edits\u6587\u4ef6\u5728\u5ba2\u6237\u7aef\u5bf9hdfs\u8fdb\u884c\u64cd\u4f5c\u65f6\u4ea7\u751f,\u5f53\u6709\u5143\u6570\u636e\u7684\u589e\u5220\u6539\u67e5\u65e5\u5fd7\u4ea7\u751f\u65f6 \u5b83\u4f1a\u76f4\u63a5\u4fdd\u5b58\u5230\u4e00\u4e2a\u5185\u90e8\u7684\u96c6\u7fa4\u4e2d\u79f0\u4e4b\u4e3aJournalNode,\u5411\u591a\u6570JournalNode\u4e0a\u4f20\u6210\u529f\u624d\u7b97\u6210\u529f\uff0c\u540c\u65f6fsimage\u6587\u4ef6\u548cedits\u6587\u4ef6\u7684\u5408\u5e76\u5de5\u4f5c\u4e5f\u7531JournalNode\u5b8c\u6210\uff0c\u4e3bNameNode\uff08Active\uff09\u6302\u4e86\u4e4b\u540e\uff0c\u5907NameNode\uff08Standby\uff09\u63a5\u7ba1\u540e\uff0c\u4f1a\u540c\u6837\u4f1a\u628a\u65e5\u5fd7\u4fdd\u5b58\u5230JournalNode\u4e2d\u3002JournalNode\u5728\u5408\u5e76fsimage\u548cedits\u6587\u4ef6\u4ee5\u66f4\u65b0fsimage\u6587\u4ef6\u65f6 \u9700\u8981\u540c\u65f6\u5408\u5e76\u4e24\u4e2aNameNode\u7684fsimage \u4ece\u800c\u4fdd\u8bc1\u77ac\u95f4\u5b9e\u73b0\u63a5\u7ba1\u3002\u6b64\u65f6\u5c31\u4e0d\u518d\u9700\u8981secondarynamenode\u4e86\uff0c\u56e0\u4e3a\u4f1a\u51fa\u73b0\u62a5\u9519\u3002<br \/>\n<code>HDFS Federation\uff08\u8054\u90a6\uff09<\/code>\uff1a\u5b83\u53ef\u4ee5\u6c34\u5e73\u6269\u5c55 \u652f\u6301\u591a\u4e2aNameNode \u6bcf\u4e2aNameNode\u5206\u7ba1\u4e00\u90e8\u5206\u76ee\u5f55\uff0c\u6240\u6709DataNode\u5171\u4eab\u6240\u6709NameNode\u7684\u8d44\u6e90\u3002Federation\u4f7f\u7528\u4e86\u591a\u4e2a\u72ec\u7acb\u7684Namenode\/NameSpace\u3002\u8fd9\u4e9bNamenode\u4e4b\u95f4\u662f\u8054\u5408\u7684\uff0c\u4e5f\u5c31\u662f\u8bf4\uff0c\u4ed6\u4eec\u4e4b\u95f4\u76f8\u4e92\u72ec\u7acb\u4e14\u4e0d\u9700\u8981\u4e92\u76f8\u534f\u8c03\uff0c\u5404\u81ea\u5206\u5de5\uff0c\u7ba1\u7406\u81ea\u5df1\u7684\u533a\u57df\u3002\u5206\u5e03\u5f0f\u7684datanode\u88ab\u7528\u4f5c\u901a\u7528\u7684\u6570\u636e\u5757\u5b58\u50a8\u5b58\u50a8\u8bbe\u5907\u3002\u6bcf\u4e2aDataNode\u8981\u5411\u96c6\u7fa4\u4e2d\u6240\u6709\u7684namenode\u6ce8\u518c\uff0c\u4e14\u5468\u671f\u6027\u7684\u5411\u6240\u6709namenode\u53d1\u9001\u5fc3\u8df3\u548c\u5757\u62a5\u544a\uff0c\u5e76\u6267\u884c\u6765\u81ea\u6240\u6709namenode\u7684\u547d\u4ee4\u3002<\/p>\n<blockquote><p>\n  \u600e\u4e48\u89e3\u51b3\u6570\u636e\u503e\u659c\u7684\u95ee\u9898\n<\/p><\/blockquote>\n<p>\u9996\u5148\u6211\u4eec\u660e\u786e\u4e00\u70b9\uff0c\u4ec0\u4e48\u662f\u6570\u636e\u503e\u659c\uff0c\u5728webUI\u4e0a\u53ef\u4ee5\u770b\u5230datanoded\u7684\u78c1\u76d8\u5360\u7528\u60c5\u51b5\uff0c\u90a3\u4e48\u6570\u636e\u503e\u659c\u5c31\u662f\u4e2a\u522b\u8282\u70b9\u78c1\u76d8\u5360\u7528\u7387\u8fdc\u8fdc\u9ad8\u4e8e\u5176\u4ed6\u8282\u70b9\uff0c\u90a3\u4e48\u5bfc\u81f4\u8fd9\u79cd\u60c5\u51b5\u53d1\u751f\u6709\u4e8c\u79cd\uff1a1\uff0c\u589e\u52a0\u4e86\u65b0\u7684\u8282\u70b9\uff0c2\uff0c\u67d0\u51e0\u4e2a\u8282\u70b9\u6302\u8f7d\u4e86\u65b0\u7684\u78c1\u76d8\uff0c\u5c31\u662f\u8f93\u65b0\u76d8\u4e0a\u5b58\u7684\u6570\u636e\u8fd8\u4e0d\u591a\uff0c\u90a3\u4e48\u8fd9\u4f1a\u7275\u626f\u51fa\u4e09\u4e2a\u95ee\u9898\uff0chdfs\u600e\u4e48\u505a\u8d1f\u8f7d\u5747\u8861\uff0c\u4ec0\u4e48\u6837\u60c5\u51b5\u4e0b\u89e6\u53d1rebalance\uff0cmapreduce\u8ba1\u7b97\u8fc7\u7a0b\u4e2d\u67d0\u4e2a\u8282\u70b9\u8ba1\u7b97\u538b\u529b\u589e\u52a0\u5f71\u54cd\u6027\u80fd\u3002<\/p>\n<p>1\u3001\u63d0\u9ad8dfs.datanode.du.reserved\u7684\u503c;\u4e5f\u5c31\u610f\u5473\u7740\u5373\u5c06\u8d85\u8fc7\u8fd9\u4e2a\u9884\u7559\u503c\u7684\u65f6\u5019\u5c31\u4f1a\u5f80\u522b\u7684\u8282\u70b9\u4e0a\u5199\u6570\u636e<\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">#100MB\n&lt;property&gt; \n     &lt;name&gt;dfs.datanode.du.reserved&lt;\/name&gt; \n     &lt;value&gt;104857600&lt;\/value&gt; \n&lt;\/property&gt;\n<\/code><\/pre>\n<p>2\u3001\u5173\u95ed\u9ad8\u78c1\u76d8\u8282\u70b9\u7684nodemanager;\u5982\u679c\u5728\u8be5\u8282\u70b9\u4e0a\u8fdb\u884c\u8ba1\u7b97\u7684\u8bdd\uff0c\u6570\u636e\u5b58\u50a8\u9996\u5148\u4f1a\u5f80\u672c\u5730\u5199\u4e00\u4efd\uff0c\u8fd9\u6837\u66f4\u52a0\u52a0\u91cd\u4e86\u672c\u5730\u8282\u70b9\u7684\u8d1f\u62c5<br \/>\n3\u3001\u964d\u4f4e\u526f\u672c\u6570;<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 opt]# hadoop dfs -setrep -w 2  \/logs\nDEPRECATED: Use of this script to execute hdfs command is deprecated.\nInstead use the hdfs command for it.\nReplication 2 set: \/logs\/Different.java\nReplication 2 set: \/logs\/GenericArrayList.java\nWaiting for \/logs\/Different.java ...............\n<\/code><\/pre>\n<p>4\u3001\u542f\u52a8hadoop\u81ea\u5e26\u7684balance;<br \/>\n\u67e5\u770b\u4f7f\u7528\u5e2e\u52a9<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 ~]# hdfs balancer -help\nUsage: hdfs balancer\n        [-policy &lt;policy&gt;]      the balancing policy: datanode or blockpool\n        [-threshold &lt;threshold&gt;]        Percentage of disk capacity\n        [-exclude [-f &lt;hosts-file&gt; | &lt;comma-separated list of hosts&gt;]]  Excludes the specified datanodes.\n        [-include [-f &lt;hosts-file&gt; | &lt;comma-separated list of hosts&gt;]]  Includes only the specified datanodes.\n        [-idleiterations &lt;idleiterations&gt;]      Number of consecutive idle iterations (-1 for Infinite) before exit.\n        [-runDuringUpgrade]     Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.\n<\/code><\/pre>\n<p>\u6267\u884c<\/p>\n<pre><code class=\"language-bash line-numbers\">[root@wq1 ~]# start-balancer.sh -threshold 20\nstarting balancer, logging to \/opt\/hadoop-2.7.7\/logs\/hadoop-root-balancer-wq1.out\n<\/code><\/pre>\n<p>\u8c03\u6574\u5e26\u5bbd<\/p>\n<pre><code class=\"language-bash line-numbers\">hdfs dfsadmin -fs hdfs: \/\/wq1 :9000 -setBalancerBandwidth 104857600\n<\/code><\/pre>\n<h2><span class=\"ez-toc-section\" id=\"hdfs%E5%9B%9B%E5%A4%A7%E6%9C%BA%E5%88%B6\"><\/span>hdfs\u56db\u5927\u673a\u5236<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ol>\n<li>\u5fc3\u8df3\u673a\u5236<\/li>\n<\/ol>\n<p>dataNode\u4f1a\u5b9a\u65f6\u7684\u5411nameNode\u53d1\u9001\u5fc3\u8df3\u62a5\u544a\uff0c\u4ece\u800cnameNode\u83b7\u53d6\u5230datanode\u5b58\u6d3b\u72b6\u6001\u3001\u5b58\u50a8\u7684block\u5217\u8868\u4fe1\u606f<\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">#\u5fc3\u8df3\u62a5\u544a\u5468\u671f\n&lt;property&gt;\n       &lt;name&gt;dfs.heartbeat.interval&lt;\/name&gt;\n       &lt;value&gt;3&lt;\/value&gt; \/\/\u5355\u4f4d\u79d2\n&lt;\/property&gt;\n#nameNode\u5728\u63a5\u6536\u4e0d\u5230dataNode\u7684\u5fc3\u8df3\u65f6\uff0c\u6b64\u65f6\u4f1a\u5411dataNode\u4e3b\u52a8\u53d1\u9001\u68c0\u67e5\n&lt;property&gt;               \n    &lt;name&gt;dfs.namenode.heartbeat.recheck-interval&lt;\/name&gt;\n    &lt;value&gt;300000&lt;\/value&gt; \/\/\u5355\u4f4d\u6beb\u79d2\n&lt;\/property&gt;\n<\/code><\/pre>\n<ol start=\"2\">\n<li>\u5b89\u5168\u673a\u5236<\/li>\n<\/ol>\n<p>hdfs\u5728\u542f\u52a8\u7684\u65f6\u5019\uff0c\u9996\u5148\u4f1a\u8fdb\u5165\u7684\u5b89\u5168\u6a21\u5f0f\u4e2d\uff0c\u5f53\u8fbe\u5230\u89c4\u5b9a\u7684\u8981\u6c42\u65f6\uff0c\u4f1a\u9000\u51fa\u5b89\u5168\u6a21\u5f0f\u3002\u5728\u5b89\u5168\u6a21\u5f0f\u4e2d\uff0c\u4e0d\u80fd\u6267\u884c\u4efb\u4f55\u4fee\u6539\u5143\u6570\u636e\u4fe1\u606f\u7684\u64cd\u4f5c\u3002<\/p>\n<p>hdfs\u7684\u5143\u6570\u636e<\/p>\n<p>1.\u62bd\u8c61\u76ee\u5f55\u6811<br \/>\n2.\u6570\u636e\u4e0e\u5757\u7684\u5bf9\u5e94\u5173\u7cfb\uff08\u6587\u4ef6\u88ab\u5207\u5206\u6210\u591a\u5c11\u4e2a\u5757\uff09<br \/>\n3.block\u5757\u5b58\u653e\u7684\u4f4d\u7f6e\u4fe1\u606f<\/p>\n<p>\u5143\u6570\u636e\u7684\u5b58\u50a8\u4f4d\u7f6e\uff1a<\/p>\n<p>\u5185\u5b58\uff1a\u5185\u5b58\u4e2d\u5b58\u50a8\u4e86\u4e00\u4efd\u5b8c\u6574\u7684\u5143\u6570\u636e\u4fe1\u606f\uff08\u62bd\u8c61\u76ee\u5f55\u6811\u3001\u6570\u636e\u4e0e\u5757\u7684\u5bf9\u5e94\u5173\u7cfb\u3001block\u5757\u5b58\u653e\u7684\u4f4d\u7f6e\u4fe1\u606f\uff09<br \/>\n\u786c\u76d8\uff1a\u62bd\u8c61\u76ee\u5f55\u6811\u3001\u6570\u636e\u4e0e\u5757\u7684\u5bf9\u5e94\u5173\u7cfb\u3002<\/p>\n<p><code>\u5176\u4e2d\u5185\u5b58\u4e2d\u7684\u5143\u6570\u636e\u7684block\u5757\u5b58\u653e\u7684\u4f4d\u7f6e\u4fe1\u606f\uff0c\u662f\u901a\u8fc7dataNode\u5411nameNode\u6c47\u62a5\u5fc3\u8df3\u65f6\u83b7\u53d6\u7684.<br \/>\n  \u786c\u76d8\u4e2d\u7684\u5143\u6570\u636e\uff0c\u662f\u56e0\u4e3a\u5185\u5b58\u4e2d\u7684\u5143\u6570\u636e\u5728\u673a\u5668\u5b95\u673a\u65f6\u5c31\u81ea\u52a8\u6d88\u5931\uff0c\u6240\u4ee5\u9700\u8981\u5c06\u5185\u5b58\u4e2d\u7684\u5143\u6570\u636e\u6301\u4e45\u5316\u5230\u786c\u76d8<br \/>\n  \u800c\u786c\u76d8\u4e2d\u7684\u5143\u6570\u636e\u53ea\u6709\u62bd\u8c61\u76ee\u5f55\u6811\u3001\u6570\u636e\u4e0e\u5757\u7684\u5bf9\u5e94\u5173\u7cfb\uff0c\u6ca1\u6709block\u5757\u5b58\u653e\u7684\u4f4d\u7f6e\u4fe1\u606f<\/code><\/p>\n<p>\u96c6\u7fa4\u542f\u52a8\u65f6dfs.namenode.safemode.min.datanodes\uff08\u542f\u52a8\u7684dataNode\u4e2a\u6570\uff09\u4e3a0\u65f6\uff0c\u5e76\u4e14\uff0c\u6570\u636e\u5757\u7684\u6700\u5c0f\u526f\u672c\u6570dfs.namenode.replication.min\u4e3a1\u65f6\uff0c\u6b64\u65f6\u4f1a\u9000\u51fa\u5b89\u5168\u6a21\u5f0f\uff0c\u4e5f\u5c31\u662f\u8bf4\uff0c\u96c6\u7fa4\u8fbe\u5230\u4e86\u6700\u5c0f\u526f\u672c\u6570\uff0c\u5e76\u4e14\u80fd\u8fd0\u884c\u7684datanode\u8282\u70b9\u4e5f\u8fbe\u5230\u4e86\u8981\u6c42\uff0c\u6b64\u65f6\u9000\u51fa\u5b89\u5168\u6a21\u5f0f<\/p>\n<p>\u542f\u52a8\u7684dataNode\u4e2a\u6570\u4e3a0\u65f6\uff0c\u5e76\u4e14\u6240\u6709\u7684\u6570\u636e\u5757\u7684\u5b58\u8d27\u7387\u8fbe\u52300.999f\u65f6\uff0c\u96c6\u7fa4\u9000\u51fa\u5b89\u5168\u6a21\u5f0f\uff08\u526f\u672c\u6570\u8fbe\u5230\u8981\u6c42\uff09<\/p>\n<p>\u624b\u52a8\u9000\u51fa\u5b89\u5168\u6a21\u5f0f\u547d\u4ee4<code>hdfs dfsadmin -safemode leave<\/code><\/p>\n<ol start=\"3\">\n<li>\u526f\u672c\u5b58\u653e\u7b56\u7565<\/li>\n<\/ol>\n<p>\u7b2c\u4e00\u4e2a\u526f\u672c\uff0c\u653e\u7f6e\u5728\u79bb\u5ba2\u6237\u7aef\u6700\u8fd1\u7684\u90a3\u4e2a\u673a\u67b6\u7684\u4efb\u610f\u8282\u70b9\uff0c\u5982\u679c\u5ba2\u6237\u7aef\u662f\u672c\u673a\uff0c\u90a3\u5c31\u5b58\u653e\u5728\u672c\u673a\uff08\u4fdd\u8bc1\u6709\u4e00\u4e2a\u526f\u672c\u6570\uff09<br \/>\n\u7b2c\u4e8c\u4e2a\u526f\u672c\uff0c\u653e\u7f6e\u5728\u8ddf\u7b2c\u4e00\u4e2a\u526f\u672c\u4e0d\u540c\u673a\u67b6\u7684\u4efb\u610f\u8282\u70b9\u4e0a<br \/>\n\u7b2c\u4e09\u4e2a\u526f\u672c\uff0c\u653e\u7f6e\u5728\u8ddf\u7b2c\u4e8c\u4e2a\u526f\u672c\u76f8\u540c\u673a\u67b6\u7684\u4e0d\u540c\u8282\u70b9\u4e0a<\/p>\n<p><code>vim hdfs-site.xml<\/code><\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">#\u8bbe\u7f6e\u526f\u672c\u6570\n&lt;property&gt;\n     &lt;name&gt;dfs.replication&lt;\/name&gt;\n     &lt;value&gt;1&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<ol start=\"4\">\n<li>\u8d1f\u8f7d\u5747\u8861<\/li>\n<\/ol>\n<p>\u5f53\u96c6\u7fa4\u5185\u65b0\u589e\u3001\u5220\u9664\u8282\u70b9\uff0c\u6216\u8005\u67d0\u4e2a\u8282\u70b9\u673a\u5668\u5185\u786c\u76d8\u5b58\u50a8\u8fbe\u5230\u9971\u548c\u503c\u3002\u5f53\u6570\u636e\u4e0d\u5e73\u8861\u65f6\uff0cMap\u4efb\u52a1\u53ef\u80fd\u4f1a\u5206\u914d\u5230\u6ca1\u6709\u5b58\u50a8\u6570\u636e\u7684\u673a\u5668\uff0c\u8fd9\u5c06\u5bfc\u81f4\u7f51\u7edc\u5e26\u5bbd\u7684\u6d88\u8017\uff0c\u4e5f\u65e0\u6cd5\u5f88\u597d\u7684\u8fdb\u884c\u672c\u5730\u8ba1\u7b97\u3002<br \/>\n\u5f53HDFS\u8d1f\u8f7d\u4e0d\u5747\u8861\u65f6\uff0c\u9700\u8981\u5bf9HDFS\u8fdb\u884c\u6570\u636e\u7684\u8d1f\u8f7d\u5747\u8861\u8c03\u6574\uff0c\u5373\u5bf9\u5404\u8282\u70b9\u673a\u5668\u4e0a\u6570\u636e\u7684\u5b58\u50a8\u5206\u5e03\u8fdb\u884c\u8c03\u6574\u3002\u4ece\u800c\uff0c\u8ba9\u6570\u636e\u5747\u5300\u7684\u5206\u5e03\u5728\u5404\u4e2aDataNode\u4e0a\uff0c\u5747\u8861IO\u6027\u80fd\u3002\u4e0b\u9762\u7684rebalance\u547d\u4ee4\u5c31\u53ef\u4ee5\u5b9e\u73b0\u8d1f\u8f7d\u5747\u8861<\/p>\n<p>\u8d1f\u8f7d\u5747\u8861\u6b65\u9aa4<\/p>\n<ul>\n<li>\u6570\u636e\u5747\u8861\u670d\u52a1\uff08Rebalancing Server\uff09\u9996\u5148\u8981\u6c42 NameNode \u751f\u6210 DataNode \u6570\u636e\u5206\u5e03\u5206\u6790\u62a5\u544a,\u83b7\u53d6\u6bcf\u4e2aDataNode\u78c1\u76d8\u4f7f\u7528\u60c5\u51b5<\/li>\n<li>Rebalancing Server\u6c47\u603b\u9700\u8981\u79fb\u52a8\u7684\u6570\u636e\u5206\u5e03\u60c5\u51b5\uff0c\u8ba1\u7b97\u5177\u4f53\u6570\u636e\u5757\u8fc1\u79fb\u8def\u7ebf\u56fe\u3002\u6570\u636e\u5757\u8fc1\u79fb\u8def\u7ebf\u56fe\uff0c\u786e\u4fdd\u7f51\u7edc\u5185\u6700\u77ed\u8def\u5f84<\/li>\n<li>\u5f00\u59cb\u6570\u636e\u5757\u8fc1\u79fb\u4efb\u52a1\uff0cProxy Source Data Node\u590d\u5236\u4e00\u5757\u9700\u8981\u79fb\u52a8\u6570\u636e\u5757<\/li>\n<li>\u5c06\u590d\u5236\u7684\u6570\u636e\u5757\u590d\u5236\u5230\u76ee\u6807DataNode\u4e0a<\/li>\n<li>\u5220\u9664\u539f\u59cb\u6570\u636e\u5757<\/li>\n<li>\u76ee\u6807DataNode\u5411Proxy Source Data Node\u786e\u8ba4\u8be5\u6570\u636e\u5757\u8fc1\u79fb\u5b8c\u6210<\/li>\n<li>Proxy Source Data Node\u5411Rebalancing Server\u786e\u8ba4\u672c\u6b21\u6570\u636e\u5757\u8fc1\u79fb\u5b8c\u6210\u3002\u7136\u540e\u7ee7\u7eed\u6267\u884c\u8fd9\u4e2a\u8fc7\u7a0b\uff0c\u76f4\u81f3\u96c6\u7fa4\u8fbe\u5230\u6570\u636e\u5747\u8861\u6807\u51c6<\/li>\n<\/ul>\n<p>hdfs\u81ea\u5e26\u6709\u811a\u672c<\/p>\n<pre><code class=\"language-bash line-numbers\">#\u8282\u70b9\u6700\u5927\u5360\u7528\u7387\u4e0e\u8282\u70b9\u7684\u6700\u5c0f\u7684\u5360\u7528\u7387\u4e4b\u95f4\u7684\u5dee\u503c\nbin\/start-balancer.sh \u2013threshold\n-threshold\n\u9ed8\u8ba4\u8bbe\u7f6e\uff1a10\uff0c\u53c2\u6570\u53d6\u503c\u8303\u56f4\uff1a0-100;\u767e\u5206\u6bd4\n\u53c2\u6570\u542b\u4e49\uff1a\u5224\u65ad\u96c6\u7fa4\u662f\u5426\u5e73\u8861\u7684\u9608\u503c\u3002\u7406\u8bba\u4e0a\uff0c\u8be5\u53c2\u6570\u8bbe\u7f6e\u7684\u8d8a\u5c0f\uff0c\u6574\u4e2a\u96c6\u7fa4\u5c31\u8d8a\u5e73\u8861\n<\/code><\/pre>\n<p>\u8fc1\u79fb\u6570\u636e\u65f6\u6240\u7528\u7f51\u7edc\u5e26\u5bbd<\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">#1M\n&lt;property&gt;\n    &lt;name&gt;dfs.balance.bandwidthPerSec&lt;\/name&gt;\n    &lt;value&gt;1048576&lt;\/value&gt;\n    &lt;description&gt; Specifies the maximum bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. &lt;\/description&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<h2><span class=\"ez-toc-section\" id=\"HDFS_HA%E6%A8%A1%E5%BC%8F%EF%BC%88%E9%AB%98%E5%8F%AF%E7%94%A8%E6%A8%A1%E5%BC%8F%EF%BC%89\"><\/span>HDFS HA\u6a21\u5f0f\uff08\u9ad8\u53ef\u7528\u6a21\u5f0f\uff09<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li>HDFS\u96c6\u7fa4\u4e2d\u53ea\u6709\u4e00\u4e2aNamenode,\u5982\u679cNamenode\u6545\u969c\uff0c\u90a3\u4e48\u8fd9\u4e2a\u96c6\u7fa4\u5c06\u4e0d\u53ef\u7528\uff0c\u76f4\u5230Namenode\u91cd\u542f\u6216\u8005\u5176\u4ed6Namenode\u63a5\u5165\u3002<\/li>\n<\/ul>\n<p>HDFS HA\u7279\u6027\u5373\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\uff0c\u5b83\u901a\u8fc7\u5728\u96c6\u7fa4\u4e2d\u540c\u65f6\u8fd0\u884c2\u4e2a(redundant)Namenodes\uff0c\u5e76\u8ba9active\u548cstandby\u4e4b\u95f4\u70ed\u5907(hot standby)\u3002\u5f53Active Namenode\u6545\u969c\u5931\u6548\u540e\uff0c\u5373\u53ef\u5feb\u901f\u6545\u969c\u8f6c\u79fb\u5230\u65b0\u7684Namenode\u4e0a(standby Namenode)\uff1b<br \/>\n\u4e3e\u4e2a\u53ef\u80fd\u4e0d\u6070\u5f53\u7684\u4f8b\u5b50\u6765\u8bf4\uff1a\u8fd9\u5c31\u597d\u6bd4\u4f60\u6709\u4e00\u6b65\u624b\u673a,\u5982\u679c\u4f60\u53ea\u63d2\u4e86\u4e00\u5f20SIM\u5361\uff0c\u4f60\u5230\u67d0\u4e00\u533a\u57df\u540e\u6ca1\u6709\u4fe1\u53f7\u540e\uff0c\u90a3\u4e48\u4f60\u5c31\u4e0d\u80fd\u4e0a\u7f51\u4e86\uff1b\u4f46\u662f\u5982\u679c\u4f60\u5206\u522b\u6709\u79fb\u52a8\u8054\u901a\u4e24\u5f20\u5361\uff0c\u5982\u679c\u5230\u67d0\u4e00\u5730\u533a\uff0c\u79fb\u52a8\u6ca1\u4fe1\u53f7\u5c31\u6362\u8054\u901a\uff0c\u4f46\u662f\u4f60\u60f3\u4e0a\u7f51\u5c31\u53ea\u80fd\u540c\u65f6\u7528\u4e00\u4e2a\u8fd0\u8425\u5546\uff0c\u624b\u673a\u8fd8\u8981\u505a\u7684\u5c31\u662f\u4f60\u8981\u5207\u6362\u8fd9\u4e24\u4e2a\u8fd0\u8425\u5546\u7684\u65f6\u5019\u8fd8\u8981\u7279\u522b\u7684\u5feb\uff0c\u4e0d\u7136\u5f71\u54cd\u4f60\u7684\u4f53\u9a8c\uff1bhdfs\u7684\u9ad8\u53ef\u7528\u6a21\u5f0f\u5c31\u8981\u4fdd\u8bc1\u4e24\u4ef6\u4e8b\uff0c\u600e\u4e48\u4fdd\u8bc1\u540c\u4e00\u65f6\u523b\u53ea\u6709\u4e00\u4e2anamenode\u5728\u63a7\u5236\u7740\u96c6\u7fa4\uff0c\u4ee5\u53ca\u4e3bnamenode\u6302\u6389\u4e4b\u540e\u5907namenode\u5feb\u901f\u63a5\u66ff\u4e3bnamenode\u3002<\/p>\n<blockquote><p>\n  hdfs HA\u6a21\u5f0f\u5404\u7ec4\u4ef6\u5173\u7cfb\u56fe\n<\/p><\/blockquote>\n<p><img src=\"https:\/\/img-blog.csdnimg.cn\/20201111012634448.png\" alt=\"\" \/><\/p>\n<blockquote><p>\n  \u5171\u4eab\u5b58\u50a8\u7cfb\u7edfQJM\uff08\u4e00\u4e2anamenode \u6302\u6389\u540e\u53e6\u4e00\u4e2anamenode\u600e\u4e48\u5feb\u901f\u83b7\u5f97\u5b83\u7684\u4fe1\u606f\uff09\n<\/p><\/blockquote>\n<ol>\n<li>\u5411jn\u5199edits\u6587\u4ef6\u8fc7\u7a0b<\/li>\n<\/ol>\n<p>\u5199\u672c\u5730\u7531\u914d\u7f6e\u4e2d\u53c2\u6570<code>dfs.namenode.name.dir<\/code>\u63a7\u5236\uff0c\u5199JN\u7531\u53c2\u6570<code>dfs.namenode.shared.edits.dir<\/code>\u63a7\u5236\uff0c\u5728\u5199EditLog\u65f6\u4f1a\u7531\u4e24\u4e2a\u4e0d\u540c\u7684\u8f93\u51fa\u6d41\u6765\u63a7\u5236\u65e5\u5fd7\u7684\u5199\u8fc7\u7a0b\uff0c\u5206\u522b\u4e3a\uff1aEditLogFileOutputStream(\u672c\u5730\u8f93\u51fa\u6d41)\u548cQuorumOutputStream(JN\u8f93\u51fa\u6d41)\u3002\u5199EditLog\u4e5f\u4e0d\u662f\u76f4\u63a5\u5199\u5230\u78c1\u76d8\u4e2d\uff0c\u4e3a\u4fdd\u8bc1\u9ad8\u541e\u5410\uff0cNameNode\u4f1a\u5206\u522b\u4e3aEditLogFileOutputStream\u548cQuorumOutputStream\u5b9a\u4e49\u4e24\u4e2a\u540c\u7b49\u5927\u5c0f\u7684Buffer\uff0c\u5927\u5c0f\u5927\u6982\u662f512KB\uff0c\u4e00\u4e2a\u5199Buffer(buffCurrent)\uff0c\u4e00\u4e2a\u540c\u6b65Buffer(buffReady)\uff0c\u8fd9\u6837\u53ef\u4ee5\u4e00\u8fb9\u5199\u4e00\u8fb9\u540c\u6b65\uff0c\u6240\u4ee5EditLog\u662f\u4e00\u4e2a\u5f02\u6b65\u5199\u8fc7\u7a0b\uff0c\u540c\u65f6\u4e5f\u662f\u4e00\u4e2a\u6279\u91cf\u540c\u6b65\u7684\u8fc7\u7a0b\uff0c\u907f\u514d\u6bcf\u5199\u4e00\u7b14\u5c31\u540c\u6b65\u4e00\u6b21\u65e5\u5fd7\u3002<\/p>\n<p>\u8fd9\u4e2a\u662f\u600e\u4e48\u5b9e\u73b0\u8fb9\u5199\u8fb9\u540c\u6b65\u7684\u5462\uff0c\u8fd9\u4e2d\u95f4\u5176\u5b9e\u662f\u6709\u4e00\u4e2a\u7f13\u51b2\u533a\u4ea4\u6362\u7684\u8fc7\u7a0b\uff0c\u5373bufferCurrent\u548cbuffReady\u5728\u8fbe\u5230\u6761\u4ef6\u65f6\u4f1a\u89e6\u53d1\u4ea4\u6362\uff0c\u5982bufferCurrent\u5728\u8fbe\u5230\u9608\u503c\u540c\u65f6bufferReady\u7684\u6570\u636e\u53c8\u540c\u6b65\u5b8c\u65f6\uff0cbufferReady\u6570\u636e\u4f1a\u6e05\u7a7a\uff0c\u540c\u65f6\u4f1a\u5c06bufferCurrent\u6307\u9488\u6307\u5411bufferReady\u4ee5\u6ee1\u8db3\u7ee7\u7eed\u5199\uff0c\u53e6\u5916\u4f1a\u5c06bufferReady\u6307\u9488\u6307\u5411bufferCurrent\u4ee5\u63d0\u4f9b\u7ee7\u7eed\u540c\u6b65EditLog<\/p>\n<ol start=\"2\">\n<li>\u4fdd\u8bc1jn\u8282\u70b9\u4e0a\u6587\u4ef6\u7684\u4e00\u81f4\u6027<\/li>\n<\/ol>\n<p><strong>2.1\u9694\u79bb\u53cc\u5199<\/strong><br \/>\n<code>\u8fd9\u6837\u5c31\u80fd\u4fdd\u8bc1\u4e3b\u5907NN\u53d1\u751f\u5207\u6362\u65f6\uff0c\u5c31\u7b97\u540c\u65f6\u5411JN\u540c\u6b65\u65e5\u5fd7\uff0c\u4e5f\u80fd\u4fdd\u8bc1\u65e5\u5fd7\u4e0d\u4f1a\u5199\u4e71\uff0c\u56e0\u4e3a\u53d1\u751f\u5207\u6362\u540e\uff0c\u539fANN\u7684EpochNumber\u80af\u5b9a\u662f\u5c0f\u4e8e\u65b0ANN\u7684EpochNumber\uff0c\u6240\u4ee5\u539fANN\u5411JN\u7684\u53d1\u8d77\u7684\u6240\u6709\u540c\u6b65\u8bf7\u6c42\u90fd\u4f1a\u62d2\u7edd\uff0c\u5b9e\u73b0\u9694\u79bb\u529f\u80fd\uff0c\u9632\u6b62\u4e86\u8111\u88c2<\/code>\u3002<br \/>\n\u5728ANN\u6bcf\u6b21\u540c\u6b65EditLog\u5230JN\u65f6\uff0c\u5148\u8981\u4fdd\u8bc1\u4e0d\u4f1a\u6709\u4e24\u4e2aNN\u540c\u65f6\u5411JN\u540c\u6b65\u65e5\u5fd7\u3002\u8fd9\u4e2a\u9694\u79bb\u662f\u600e\u4e48\u505a\u7684\u3002\u8fd9\u91cc\u9762\u6d89\u53ca\u4e00\u4e2a\u5f88\u91cd\u8981\u7684\u6982\u5ff5Epoch Numbers\uff0c\u5f88\u591a\u5206\u5e03\u5f0f\u7cfb\u7edf\u90fd\u4f1a\u7528\u5230\u3002Epoch\u6709\u5982\u4e0b\u51e0\u4e2a\u7279\u6027\uff1a<\/p>\n<p>\u5f53NN\u6210\u4e3a\u6d3b\u52a8\u7ed3\u70b9\u65f6\uff0c\u5176\u4f1a\u88ab\u8d4b\u4e88\u4e00\u4e2aEpochNumber<br \/>\n\u6bcf\u4e2aEpochNumber\u662f\u60df\u4e00\u7684\uff0c\u4e0d\u4f1a\u6709\u76f8\u540c\u7684EpochNumber\u51fa\u73b0<br \/>\nEpochNumber\u6709\u4e25\u683c\u987a\u5e8f\u4fdd\u8bc1\uff0c\u6bcf\u6b21NN\u5207\u6362\u540e\u5176EpochNumber\u90fd\u4f1a\u81ea\u589e1\uff0c\u540e\u9762\u751f\u6210\u7684EpochNumber\u90fd\u4f1a\u5927\u4e8e\u524d\u9762\u7684EpochNumber\uff0cQJM\u662f\u600e\u4e48\u4fdd\u8bc1\u4e0a\u9762\u7279\u6027\u7684\u5462\uff0c\u4e3b\u8981\u6709\u4ee5\u4e0b\u51e0\u70b9\uff1a<\/p>\n<ul>\n<li>\u7b2c\u4e00\u6b65\uff0c\u5728\u5bf9EditLog\u4f5c\u4efb\u4f55\u4fee\u6539\u524d\uff0cQuorumJournalManager(NameNode\u4e0a)\u5fc5\u987b\u88ab\u8d4b\u4e88\u4e00\u4e2aEpochNumber<\/li>\n<li>\u7b2c\u4e8c\u6b65\uff0c QJM\u628a\u81ea\u5df1\u7684EpochNumber\u901a\u8fc7newEpoch(N)\u7684\u65b9\u5f0f\u53d1\u9001\u7ed9\u6240\u6709JN\u7ed3\u70b9<\/li>\n<li>\u7b2c\u4e09\u6b65\uff0c \u5f53JN\u6536\u5230newEpoch\u8bf7\u6c42\u540e\uff0c\u4f1a\u628aQJM\u7684EpochNumber\u4fdd\u5b58\u5230\u4e00\u4e2alastPromisedEpoch\u53d8\u91cf\u4e2d\u5e76\u6301\u4e45\u5316\u5230\u672c\u5730\u78c1\u76d8<\/li>\n<li>\u7b2c\u56db\u6b65\uff0c ANN\u540c\u6b65\u65e5\u5fd7\u5230JN\u7684\u4efb\u4f55RPC\u8bf7\u6c42\uff08\u5982logEdits(),startLogSegment()\u7b49\uff09\uff0c\u90fd\u5fc5\u987b\u5305\u542bANN\u7684EpochNumber<\/li>\n<li>\u7b2c\u4e94\u6b65\uff0cJN\u5728\u6536\u5230RPC\u8bf7\u6c42\u540e\uff0c\u4f1a\u5c06\u4e4b\u4e0elastPromisedEpoch\u5bf9\u6bd4\uff0c\u5982\u679c\u8bf7\u6c42\u7684EpochNumber\u5c0f\u4e8elastPromisedEpoch,\u5c06\u4f1a\u62d2\u7edd\u540c\u6b65\u8bf7\u6c42\uff0c\u53cd\u4e4b\uff0c\u4f1a\u63a5\u53d7\u540c\u6b65\u8bf7\u6c42\u5e76\u5c06\u8bf7\u6c42\u7684EpochNumber\u4fdd\u5b58\u5728lastPromisedEpoch<\/li>\n<\/ul>\n<p><strong>2.2\u6062\u590din-process\u65e5\u5fd7<\/strong><br \/>\n<code>\u5982\u679c\u5728\u5199\u8fc7\u7a0b\u4e2d\u5199\u5931\u8d25\u4e86\uff0c\u53ef\u80fd\u5404\u4e2aJN\u4e0a\u7684EditLog\u7684\u957f\u5ea6\u90fd\u4e0d\u4e00\u6837\uff0c\u9700\u8981\u5728\u5f00\u59cb\u5199\u4e4b\u524d\u5c06\u4e0d\u4e00\u81f4\u7684\u90e8\u5206\u6062\u590d<\/code><\/p>\n<ol start=\"3\">\n<li>\u4ecejn\u8bfb\u53d6\u6587\u4ef6\u5bf9\u8fc7\u7a0b<\/li>\n<\/ol>\n<p>\u5728SNN\u4e0a\u5148\u68c0\u67e5\u524d\u7f6e\u6761\u4ef6\uff0c\u524d\u7f6e\u6761\u4ef6\u5305\u62ec\u4e24\u4e2a\u65b9\u9762\uff1a\u8ddd\u79bb\u4e0a\u6b21Checkpointing\u7684\u65f6\u95f4\u95f4\u9694\u548cEditLog\u4e2d\u4e8b\u52a1\u6761\u6570\u9650\u5236\u3002\u524d\u7f6e\u6761\u4ef6\u4efb\u4f55\u4e00\u4e2a\u6ee1\u8db3\u90fd\u4f1a\u89e6\u53d1Checkpointing\uff0c\u7136\u540eSNN\u4f1a\u5c06\u6700\u65b0\u7684NameSpace\u6570\u636e\u5373SNN\u5185\u5b58\u4e2d\u5f53\u524d\u72b6\u6001\u7684\u5143\u6570\u636e\u4fdd\u5b58\u5230\u4e00\u4e2a\u4e34\u65f6\u7684fsimage\u6587\u4ef6( fsimage.ckpt\uff09\u7136\u540e\u6bd4\u5bf9\u4eceJN\u4e0a\u62c9\u5230\u7684\u6700\u65b0EditLog\u7684\u4e8b\u52a1ID\uff0c\u5c06fsimage.ckpt_\u4e2d\u6ca1\u6709\uff0cEditLog\u4e2d\u6709\u7684\u6240\u6709\u5143\u6570\u636e\u4fee\u6539\u8bb0\u5f55\u5408\u5e76\u4e00\u8d77\u5e76\u91cd\u547d\u540d\u6210\u65b0\u7684fsimage\u6587\u4ef6\uff0c\u540c\u65f6\u751f\u6210\u4e00\u4e2amd5\u6587\u4ef6\u3002\u5c06\u6700\u65b0\u7684fsimage\u518d\u901a\u8fc7HTTP\u8bf7\u6c42\u4f20\u56deANN\u3002<\/p>\n<blockquote><p>\n  \u9009\u4e3e\u51faleader\n<\/p><\/blockquote>\n<p><img src=\"https:\/\/img-blog.csdnimg.cn\/20201105172304933.png\" alt=\"HA\" title=\"HA\" \/><br \/>\n\u00a0\u00a01\uff09HealthMonitor\u521d\u59cb\u5316\u5b8c\u6210\u540e\u901a\u8fc7\u5185\u90e8\u7ebf\u7a0b\u8c03\u7528NameNode\u7684RPC\u63a5\u53e3\u5bf9\u5176\u8fdb\u884c\u5065\u5eb7\u68c0\u67e5<br \/>\n\u00a0 2\uff09\u5982\u679c\u68c0\u67e5\u5230NameNode\u72b6\u6001\u5f02\u5e38,\u4f1a\u56de\u8c03ZKFailoverContorller\u6ce8\u518c\u7684\u56de\u8c03\u51fd\u6570\u8fdb\u884c\u76f8\u5e94\u7684\u5904\u7406<br \/>\n\u00a0 3\uff09\u5982\u679cZKFailoverController\u53d1\u73b0\u96c6\u7fa4\u9700\u8981\u8fdb\u884c\u4e3b\u5907\u9009\u4e3e\uff0c\u4f1a\u4f7f\u7528ActiveStanbyElector\u548czookeeper\u96c6\u7fa4\u901a\u4fe1\u5b8c\u6210\u4e3b\u5907\u5207\u6362<br \/>\n\u00a0 4\uff09ActiveStanbyElector\u5728\u5b8c\u6210\u4e3b\u5907\u5207\u6362\u540e\uff0c\u56de\u8c03ZKFailoverController\u6ce8\u518c\u7684\u65b9\u6cd5\u4f7fNameNode\u53d8\u6210active\u6216\u8005stanby\u72b6\u6001<\/p>\n<blockquote><p>\n  \u5728\u6545\u969c\u5207\u6362\u671f\u95f4\uff0cZooKeeper\u4e3b\u8981\u662f\u53d1\u6325\u4ec0\u4e48\u4f5c\u7528\u5462\uff0c\u6709\u4ee5\u4e0b\u51e0\u70b9\uff1a\n<\/p><\/blockquote>\n<ul>\n<li>\u5931\u8d25\u4fdd\u62a4\uff1a\u96c6\u7fa4\u4e2d\u6bcf\u4e00\u4e2aNameNode\u90fd\u4f1a\u5728ZooKeeper\u7ef4\u62a4\u4e00\u4e2a\u6301\u4e45\u7684session,\u673a\u5668\u4e00\u65e6\u6302\u6389\uff0csession\u5c31\u4f1a\u8fc7\u671f\uff0c\u6545\u969c\u8fc1\u79fb\u5c31\u4f1a\u89e6\u53d1<\/li>\n<li>Active NameNode\u9009\u62e9\uff1aZooKeeper\u6709\u4e00\u4e2a\u9009\u62e9ActiveNN\u7684\u673a\u5236\uff0c\u4e00\u65e6\u73b0\u6709\u7684ANN\u5b95\u673a\uff0c\u5176\u4ed6NameNode\u53ef\u4ee5\u5411ZooKeeper\u7533\u8bf7\u6392\u4ed6\u6210\u4e3a\u4e0b\u4e00\u4e2aActive\u8282\u70b9<\/p>\n<\/li>\n<li>\n<p>\u9632\u8111\u88c2\uff1a ZK\u672c\u8eab\u662f\u5f3a\u4e00\u81f4\u548c\u9ad8\u53ef\u7528\u7684\uff0c\u53ef\u4ee5\u7528\u5b83\u6765\u4fdd\u8bc1\u540c\u4e00\u65f6\u523b\u53ea\u6709\u4e00\u4e2a\u6d3b\u52a8\u8282\u70b9<br \/>\n\u90a3\u5728\u54ea\u4e9b\u573a\u666f\u4f1a\u89e6\u53d1\u81ea\u52a8\u5207\u6362\u5462\uff0c\u4eceHDFS-2185\u4e2d\u5f52\u7eb3\u4e86\u4ee5\u4e0b\u51e0\u4e2a\u573a\u666f\uff1a<\/p>\n<\/li>\n<li>\n<p>ActiveNN JVM\u5954\u6e83\uff1aANN\u4e0aHealthMonitor\u72b6\u6001\u4e0a\u62a5\u4f1a\u6709\u8fde\u63a5\u8d85\u65f6\u5f02\u5e38\uff0cHealthMonitor\u4f1a\u89e6\u53d1\u72b6\u6001\u8fc1\u79fb\u81f3SERVICE_NOT_RESPONDING, \u7136\u540eANN\u4e0a\u7684ZKFC\u4f1a\u9000\u51fa\u9009\u4e3e\uff0cSNN\u4e0a\u7684ZKFC\u4f1a\u83b7\u5f97Active Lock, \u4f5c\u76f8\u5e94\u9694\u79bb\u540e\u6210\u4e3aActive\u7ed3\u70b9\u3002<\/p>\n<\/li>\n<li>ActiveNN JVM\u51bb\u7ed3\uff1a\u8fd9\u4e2a\u662fJVM\u6ca1\u5954\u6e83\uff0c\u4f46\u4e5f\u65e0\u6cd5\u54cd\u5e94\uff0c\u540c\u5954\u6e83\u4e00\u6837\uff0c\u4f1a\u89e6\u53d1\u81ea\u52a8\u5207\u6362\u3002<\/li>\n<li>ActiveNN \u673a\u5668\u5b95\u673a\uff1a\u6b64\u65f6ActiveStandbyElector\u4f1a\u5931\u53bb\u540cZK\u7684\u5fc3\u8df3\uff0c\u4f1a\u8bdd\u8d85\u65f6\uff0cSNN\u4e0a\u7684ZKFC\u4f1a\u901a\u77e5ZK\u5220\u9664ANN\u7684\u6d3b\u52a8\u9501\uff0c\u4f5c\u76f8\u5e94\u9694\u79bb\u540e\u5b8c\u6210\u4e3b\u5907\u5207\u6362\u3002<\/li>\n<li>ActiveNN \u5065\u5eb7\u72b6\u6001\u5f02\u5e38\uff1a \u6b64\u65f6HealthMonitor\u4f1a\u6536\u5230\u4e00\u4e2aHealthCheckFailedException\uff0c\u5e76\u89e6\u53d1\u81ea\u52a8\u5207\u6362\u3002<\/li>\n<li>Active ZKFC\u5954\u6e83\uff1a\u867d\u7136ZKFC\u662f\u4e00\u4e2a\u72ec\u7acb\u7684\u8fdb\u7a0b\uff0c\u4f46\u56e0\u8bbe\u8ba1\u7b80\u5355\u4e5f\u5bb9\u6613\u51fa\u95ee\u9898\uff0c\u4e00\u65e6ZKFC\u8fdb\u7a0b\u6302\u6389\uff0c\u867d\u7136\u6b64\u65f6NameNode\u662fOK\u7684\uff0c\u4f46\u7cfb\u7edf\u4e5f\u8ba4\u4e3a\u9700\u8981\u5207\u6362\uff0c\u6b64\u65f6SNN\u4f1a\u53d1\u4e00\u4e2a\u8bf7\u6c42\u5230ANN\u8981\u6c42ANN\u653e\u5f03\u4e3b\u7ed3\u70b9\u4f4d\u7f6e\uff0cANN\u6536\u5230\u8bf7\u6c42\u540e\uff0c\u4f1a\u89e6\u53d1\u5b8c\u6210\u81ea\u52a8\u5207\u6362\u3002<\/li>\n<li>ZooKeeper\u5954\u6e83\uff1a\u5982\u679cZK\u5954\u6e83\u4e86\uff0c\u4e3b\u5907NN\u4e0a\u7684ZKFC\u90fd\u4f1a\u611f\u77e5\u65ad\u8fde\uff0c\u6b64\u65f6\u4e3b\u5907NN\u4f1a\u8fdb\u5165\u4e00\u4e2aNeutralMode\u6a21\u5f0f\uff0c\u540c\u65f6\u4e0d\u6539\u53d8\u4e3b\u5907NN\u7684\u72b6\u6001\uff0c\u7ee7\u7eed\u53d1\u6325\u4f5c\u7528\uff0c\u53ea\u4e0d\u8fc7\u6b64\u65f6\uff0c\u5982\u679cANN\u4e5f\u6545\u969c\u4e86\uff0c\u90a3\u96c6\u7fa4\u65e0\u6cd5\u53d1\u6325Failover, \u4e5f\u5c31\u4e0d\u53ef\u7528\u4e86\uff0c\u6240\u4ee5\u5bf9\u4e8e\u6b64\u79cd\u573a\u666f\uff0cZK\u4e00\u822c\u662f\u4e0d\u5141\u8bb8\u6302\u6389\u5230\u591a\u53f0\uff0c\u81f3\u5c11\u8981\u6709N\/2+1\u53f0\u4fdd\u6301\u670d\u52a1\u624d\u7b97\u662f\u5b89\u5168\u7684\u3002<\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"%E4%B8%BA%E4%BA%86%E6%9E%84%E5%BB%BAHA%E9%9B%86%E7%BE%A4%E6%9E%B6%E6%9E%84%EF%BC%8C%E4%BD%A0%E9%9C%80%E8%A6%81%E5%87%86%E5%A4%87%E5%A6%82%E4%B8%8B%E8%B5%84%E6%BA%90%EF%BC%9A\"><\/span>\u4e3a\u4e86\u6784\u5efaHA\u96c6\u7fa4\u67b6\u6784\uff0c\u4f60\u9700\u8981\u51c6\u5907\u5982\u4e0b\u8d44\u6e90\uff1a<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Namenode\u673a\u5668\uff1a\u4e24\u53f0\u914d\u7f6e\u5bf9\u7b49\u7684\u7269\u7406\u673a\u5668\uff0c\u5b83\u4eec\u5206\u522b\u8fd0\u884cActive\u548cStandby Node\u3002<\/p>\n<p>JouralNode\u673a\u5668\uff1a\u8fd0\u884cJouralNodes\u7684\u673a\u5668\u3002JouralNode\u5b88\u62a4\u8fdb\u7a0b\u76f8\u5f53\u7684\u8f7b\u91cf\u7ea7\uff0c\u5b83\u4eec\u53ef\u4ee5\u548chadoop\u7684\u5176\u4ed6\u8fdb\u7a0b\u90e8\u7f72\u5728\u4e00\u8d77\uff0c\u6bd4\u5982Namenodes\u3001jobTracker\u3001ResourceManager\u7b49\u3002\u4e0d\u8fc7\u4e3a\u4e86\u5f62\u6210\u591a\u6570\u6d3e(majority)\uff0c\u81f3\u5c11\u9700\u89813\u4e2aJouralNodes\uff0c\u56e0\u4e3aedits\u64cd\u4f5c\u5fc5\u987b\u5728\u591a\u6570\u6d3e\u4e0a\u5199\u5165\u6210\u529f\u3002\u5f53\u7136JNS\u7684\u4e2a\u6570\u53ef\u4ee5 > 3\uff0c\u4e14\u901a\u5e38\u4e3a\u5947\u6570(3,5,7)\uff0c\u8fd9\u6837\u53ef\u4ee5\u66f4\u597d\u7684\u5bb9\u9519\u548c\u5f62\u6210\u591a\u6570\u6d3e\u3002\u5982\u679c\u4f60\u8fd0\u884c\u4e86N\u4e2aJNS\uff0c\u90a3\u4e48\u5b83\u53ef\u4ee5\u5141\u8bb8(N-1)\/2\u4e2aJNS\u8fdb\u7a0b\u5931\u6548\u5e76\u4e14\u4e0d\u5f71\u54cd\u5de5\u4f5c\u3002<\/p>\n<p>\u6b64\u5916\uff0c\u5728HA\u96c6\u7fa4\u4e2d\uff0cstandby namenode\u8fd8\u4f1a\u5bf9namespace\u8fdb\u884ccheckpoint\u64cd\u4f5c(\u7ee7\u627fBackup Namenode\u7684\u7279\u6027)\uff0c\u56e0\u6b64\uff0c\u5c31\u4e0d\u9700\u8981\u5728HA\u96c6\u7fa4\u4e2d\u8fd0\u884cSecondaryNamenode\u3001CheckpointNode\u6216\u8005BackupNode\u3002\u4e8b\u5b9e\u4e0a\uff0cHA\u67b6\u6784\u4e2d\u8fd0\u884c\u4e0a\u8ff0\u8282\u70b9\uff0c\u5c06\u4f1a\u51fa\u9519(\u4e0d\u5141\u8bb8)\u3002<\/p>\n<blockquote><p>\n  Hadoop\u96c6\u7fa4\u4e2dHadoop\u90fd\u5206\u522b\u9700\u8981\u542f\u52a8\u54ea\u4e9b\u8fdb\u7a0b\uff0c\u5b83\u4eec\u7684\u4f5c\u7528\u5206\u522b\u662f\u4ec0\u4e48?\n<\/p><\/blockquote>\n<p>1\uff09NameNode\u5b83\u662fhadoop\u4e2d\u7684\u4e3b\u670d\u52a1\u5668\uff0c\u7ba1\u7406\u6587\u4ef6\u7cfb\u7edf\u540d\u79f0\u7a7a\u95f4\u548c\u5bf9\u96c6\u7fa4\u4e2d\u5b58\u50a8\u7684\u6587\u4ef6\u7684\u8bbf\u95ee\uff0c\u4fdd\u5b58\u6709metadate\u3002<br \/>\n2\uff09SecondaryNameNode\u5b83\u4e0d\u662fnamenode\u7684\u5197\u4f59\u5b88\u62a4\u8fdb\u7a0b\uff0c\u800c\u662f\u63d0\u4f9b\u5468\u671f\u68c0\u67e5\u70b9\u548c\u6e05\u7406\u4efb\u52a1\u3002\u5e2e\u52a9NN\u5408\u5e76editslog\uff0c\u51cf\u5c11NN\u542f\u52a8\u65f6\u95f4\u3002<br \/>\n3\uff09DataNode\u5b83\u8d1f\u8d23\u7ba1\u7406\u8fde\u63a5\u5230\u8282\u70b9\u7684\u5b58\u50a8\uff08\u4e00\u4e2a\u96c6\u7fa4\u4e2d\u53ef\u4ee5\u6709\u591a\u4e2a\u8282\u70b9\uff09\u3002\u6bcf\u4e2a\u5b58\u50a8\u6570\u636e\u7684\u8282\u70b9\u8fd0\u884c\u4e00\u4e2adatanode\u5b88\u62a4\u8fdb\u7a0b\u3002<br \/>\n4\uff09ResourceManager\uff08JobTracker\uff09JobTracker\u8d1f\u8d23\u8c03\u5ea6DataNode\u4e0a\u7684\u5de5\u4f5c\u3002\u6bcf\u4e2aDataNode\u6709\u4e00\u4e2aTaskTracker\uff0c\u5b83\u4eec\u6267\u884c\u5b9e\u9645\u5de5\u4f5c\u3002<br \/>\n5\uff09NodeManager\uff08TaskTracker\uff09\u6267\u884c\u4efb\u52a1<br \/>\n6\uff09DFSZKFailoverController\u9ad8\u53ef\u7528\u65f6\u5b83\u8d1f\u8d23\u76d1\u63a7NN\u7684\u72b6\u6001\uff0c\u5e76\u53ca\u65f6\u7684\u628a\u72b6\u6001\u4fe1\u606f\u5199\u5165ZK\u3002\u5b83\u901a\u8fc7\u4e00\u4e2a\u72ec\u7acb\u7ebf\u7a0b\u5468\u671f\u6027\u7684\u8c03\u7528NN\u4e0a\u7684\u4e00\u4e2a\u7279\u5b9a\u63a5\u53e3\u6765\u83b7\u53d6NN\u7684\u5065\u5eb7\u72b6\u6001\u3002FC\u4e5f\u6709\u9009\u62e9\u8c01\u4f5c\u4e3aActive NN\u7684\u6743\u5229\uff0c\u56e0\u4e3a\u6700\u591a\u53ea\u6709\u4e24\u4e2a\u8282\u70b9\uff0c\u76ee\u524d\u9009\u62e9\u7b56\u7565\u8fd8\u6bd4\u8f83\u7b80\u5355\uff08\u5148\u5230\u5148\u5f97\uff0c\u8f6e\u6362\uff09\u3002<br \/>\n7\uff09JournalNode \u9ad8\u53ef\u7528\u60c5\u51b5\u4e0b\u5b58\u653enamenode\u7684editlog\u6587\u4ef6.<\/p>\n<blockquote><p>\n  \u628a\u96c6\u7fa4\u4e0a\u7684\u914d\u7f6e\u6587\u4ef6\u53d1\u9001\u5230\u5176\u4ed6\u8282\u70b9\u4e0a\u7684shell\u811a\u672c\n<\/p><\/blockquote>\n<pre><code class=\"language-shell line-numbers\">#\u6839\u636e\u81ea\u5df1\u7684\u673a\u5668\u540d\u4fee\u6539\u540e\u518d\u4f7f\u7528\n#!\/bin\/bash\nHOSTS='wq1 wq2 wq3'\nfor HOST in <span class=\"katex math inline\">{HOSTS}\ndo\n  echo \"++\u6b63\u5728\u590d\u5236<\/span>1\u5230<span class=\"katex math inline\">HOST\u7684<\/span>2\u4e2d...\"\n  scp -rq <span class=\"katex math inline\">{1}<\/span>{HOST}:${2}\n  echo \"--\u53d1\u9001\u5b8c\u6210\uff01\"\ndone\n<\/code><\/pre>\n<blockquote><p>\n  \u4ec0\u4e48\u6837\u7684\u60c5\u51b5\u4e0b\u89e6\u53d1rebalance\n<\/p><\/blockquote>\n<p>\u5728hdfs\u96c6\u7fa4\u8fd0\u884c\u8fc7\u7a0b\u4e2d\uff0c\u4f1a\u6709\u591a\u79cd\u60c5\u51b5\u4e0b\u5bfc\u81f4\u4e0d\u540c\u673a\u5668\u4e0a\u5b58\u50a8\u7684\u6570\u636e\u91cf\u5927\u5c0f\u5dee\u522b\u8f83\u5927\uff0c\u8fd0\u884c\u8fc7\u7a0b\u4e2d\u8fdb\u884c\u589e\u52a0\u6216\u51cf\u5c11\u7684\u526f\u672c\uff0c\u6216\u589e\u52a0datanode\u8282\u70b9<br \/>\n<code>\u524d\u63d0<\/code>\uff1arebalance \u662f\u4e00\u4e2a\u975e\u81ea\u52a8\u7684\u7ba1\u7406\u529f\u80fd<br \/>\n<code>\u4e3a\u4ec0\u4e48\u8981Rebalance<\/code>\uff1a\u5404\u4e2a\u8282\u70b9\u95f4\u7684\u6570\u636e\u4e0d\u5747\u8861<br \/>\n<code>\u9700\u8981Rebalance\u7684\u51e0\u79cd\u60c5\u5f62<\/code>\uff1a\u589e\u52a0datanode\uff0c\u589e\u52a0\u78c1\u76d8\u6570\u91cf\uff0c\u4e8c\u8005\u90fd\u6709\u3002<\/p>\n<pre><code class=\"language-shell line-numbers\">#\u8be5\u547d\u4ee4\u7528\u4e8e\u9488\u5bf9\u8282\u70b9\u78c1\u76d8\u90fd\u4e0d\u53d1\u751f\u53d8\u52a8\u7684\u60c5\u51b5\u8fdb\u884c\nhdfs balancer -include wq1,wq2,wq3\nhadoop balancer \u65e0\u6cd5\u4fa6\u67e5\u5230\u8fd9\u79cd \u5355\u4e2aDataNode \u7684 disk \u4e4b\u95f4\u6570\u636e\u4e0d\u5e73\u8861\n<\/code><\/pre>\n<ol>\n<li>hadoop 2.x \u4e5f\u662f\u53ef\u4ee5\u53ef\u4ee5\u8fc2\u56de\u5b9e\u73b0 disk \u7ea7\u522b\u7684rebalance<\/li>\n<\/ol>\n<p>\u5728 hdfs-site.xml\u4e2d\u8bbe\u7f6e \u6bcf\u5757\u78c1\u76d8\u7684\u9884\u7559\u7a7a\u95f4,\u5c40\u9650\u6027\u5728\u4e8e\u5982\u679c\u4e00\u4e2aDataNode \u6302\u8f7d\u7684\u597d\u51e0\u6b3e\u4e0d\u540c\u5927\u5c0f\u7684\u78c1\u76d8\uff0c\u670950G 500G 1T 2T 3T,\u8bbe\u7f6e 30G,\u663e\u7136\u5bf950G\u5f71\u54cd\u662f\u6700\u5927\u7684<\/p>\n<pre data-language=XML><code class=\"language-markup line-numbers\">&lt;property&gt;\n&lt;name&gt;dfs.datanode.du.reserved&lt;\/name&gt;\n&lt;value&gt;82212254720&lt;\/value&gt;\n&lt;\/property&gt;\n#\u4e0b\u9762\u8fd9\u4e2a\u914d\u7f6e\u5728hadoop 2.x\u4e0d\u4f1a\u751f\u6548\uff0c\u4f46\u57283.x\u6709\u6548\n&lt;property&gt;\n&lt;name&gt;dfs.disk.balancer.enabled&lt;\/name&gt;\n&lt;value&gt;true&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<ol start=\"2\">\n<li>\u8ba9hdfs \u4f18\u5148\u5f80\u65b0\u76d8 \u5199\u5165\u6570\u636e \uff0c\u6539\u53d8 hdfs \u7684\u9ed8\u8ba4\u5199\u5165\u7b56\u7565\uff0c\u76d1\u63a7 \u5404\u4e2a\u78c1\u76d8\u5199\u5165\u5dee\u989d \u542f\u52a8\u81ea\u5e73\u8861<\/li>\n<\/ol>\n<pre data-language=XML><code class=\"language-markup line-numbers\">&lt;property&gt;\n    &lt;name&gt;dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold&lt;\/name&gt;\n    &lt;value&gt;20737418240&lt;\/value&gt; &lt;!-- 20 GB --&gt;\n&lt;\/property&gt;\n&lt;property&gt;\n    &lt;name&gt;dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction&lt;\/name&gt;\n    &lt;value&gt;1.0f&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<blockquote><p>\n  \u600e\u4e48\u5904\u7406\u5927\u91cf\u7684\u5c0f\u6587\u4ef6<br \/>\n  \u4e3a\u4ec0\u4e48\uff1a\u5927\u91cf\u5c0f\u6587\u4ef6\u5e26\u7ed9namenode\u5185\u5b58\u7684\u538b\u529b\uff0c\u8bbf\u95ee\u5927\u91cf\u5c0f\u6587\u4ef6\u901f\u5ea6\u8fdc\u8fdc\u5c0f\u4e8e\u8bbf\u95ee\u51e0\u4e2a\u5927\u6587\u4ef6\n<\/p><\/blockquote>\n<h4><span class=\"ez-toc-section\" id=\"hadoop%E8%87%AA%E5%B8%A6%E7%9A%84%E8%A7%A3%E5%86%B3%E5%B0%8F%E6%96%87%E4%BB%B6%E9%97%AE%E9%A2%98%E7%9A%84%E6%96%B9%E6%A1%88\"><\/span>hadoop\u81ea\u5e26\u7684\u89e3\u51b3\u5c0f\u6587\u4ef6\u95ee\u9898\u7684\u65b9\u6848<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>\u9996\u5148\u660e\u786e\u5c0f\u6587\u4ef6\u7684\u6765\u6e90\uff0c\u5982\u679c\u662f\u4eceflume\u91c7\u96c6\u8fc7\u6765\u7684\uff0c\u90a3\u4e48\u5b8c\u5168\u53ef\u4ee5\u8bbe\u7f6e\u53c2\u6570\u4f7f\u5176\u6587\u4ef6\u91c7\u96c6\u91cf\u8fbe\u5230\u5757\u5927\u5c0f\u518d\u4f20\u5230hdfs\u4e0a\uff1b<\/p>\n<ul>\n<li>Hadoop Archive<\/li>\n<\/ul>\n<p><code>\u7279\u70b9<\/code>\uff1aHadoop Archive\u6216\u8005HAR\uff0c\u662f\u4e00\u4e2a\u9ad8\u6548\u5730\u5c06\u5c0f\u6587\u4ef6\u653e\u5165HDFS\u5757\u4e2d\u7684\u6587\u4ef6\u5b58\u6863\u5de5\u5177\uff0c\u5b83\u80fd\u591f\u5c06\u591a\u4e2a\u5c0f\u6587\u4ef6\u6253\u5305\u6210\u4e00\u4e2aHAR\u6587\u4ef6\uff0c\u8fd9\u6837\u5728\u51cf\u5c11namenode\u5185\u5b58\u4f7f\u7528\u7684\u540c\u65f6\uff0c\u4ecd\u7136\u5141\u8bb8\u5bf9\u6587\u4ef6\u8fdb\u884c\u900f\u660e\u7684\u8bbf\u95ee\u3002<br \/>\n<code>\u9650\u5236\u56e0\u7d20<\/code>\uff1a\u4f7f\u7528HAR\u65f6\u9700\u8981\u4e24\u70b9\uff0c\u7b2c\u4e00\uff0c\u5bf9\u5c0f\u6587\u4ef6\u8fdb\u884c\u5b58\u6863\u540e\uff0c\u539f\u6587\u4ef6\u5e76\u4e0d\u4f1a\u81ea\u52a8\u88ab\u5220\u9664\uff0c\u9700\u8981\u7528\u6237\u81ea\u5df1\u5220\u9664\uff1b\u7b2c\u4e8c\uff0c\u521b\u5efaHAR\u6587\u4ef6\u7684\u8fc7\u7a0b\u5b9e\u9645\u4e0a\u662f\u5728\u8fd0\u884c\u4e00\u4e2amapreduce\u4f5c\u4e1a\uff0c\u56e0\u800c\u9700\u8981\u6709\u4e00\u4e2ahadoop\u96c6\u7fa4\u8fd0\u884c\u6b64\u547d\u4ee4\u3002<br \/>\n\u6b64\u5916\uff0cHAR\u8fd8\u6709\u4e00\u4e9b\u7f3a\u9677\uff1a\u7b2c\u4e00\uff0c\u4e00\u65e6\u521b\u5efa\uff0cArchives\u4fbf\u4e0d\u53ef\u6539\u53d8\u3002\u8981\u589e\u52a0\u6216\u79fb\u9664\u91cc\u9762\u7684\u6587\u4ef6\uff0c\u5fc5\u987b\u91cd\u65b0\u521b\u5efa\u5f52\u6863\u6587\u4ef6\u3002\u7b2c\u4e8c\uff0c\u8981\u5f52\u6863\u7684\u6587\u4ef6\u540d\u4e2d\u4e0d\u80fd\u6709\u7a7a\u683c\uff0c\u5426\u5219\u4f1a\u629b\u51fa\u5f02\u5e38\uff0c\u53ef\u4ee5\u5c06\u7a7a\u683c\u7528\u5176\u4ed6\u7b26\u53f7\u66ff\u6362(\u4f7f\u7528-Dhar.space.replacement.enable=true \u548c-Dhar.space.replacement\u53c2\u6570)<\/p>\n<ul>\n<li>Sequence file<\/li>\n<\/ul>\n<p>sequence file\u7531\u4e00\u7cfb\u5217\u7684\u4e8c\u8fdb\u5236key\/value\u7ec4\u6210\uff0c\u5982\u679c\u4e3akey\u5c0f\u6587\u4ef6\u540d\uff0cvalue\u4e3a\u6587\u4ef6\u5185\u5bb9\uff0c\u5219\u53ef\u4ee5\u5c06\u5927\u6279\u5c0f\u6587\u4ef6\u5408\u5e76\u6210\u4e00\u4e2a\u5927\u6587\u4ef6<\/p>\n<blockquote><p>\n  flume\u5728\u5411hdfs\u5199\u6587\u4ef6\u65f6\u5c0f\u6587\u4ef6\u89e3\u51b3\u529e\u6cd5\n<\/p><\/blockquote>\n<pre><code class=\"language-conf line-numbers\">a1.sinks.k1.hdfs.path = hdfs:\/\/nameservice1\/tmp\/flume\/jbw\/%y-%m-%d\/%H%M%S\n#\u8be5\u5c5e\u6027\u53ef\u4ee5\u8bbe\u7f6e\u4e3afalse\na1.sinks.k1.hdfs.round = true\na1.sinks.k1.hdfs.roundValue = 60\na1.sinks.k1.hdfs.roundUnit = minute\nrollSize\n\u9ed8\u8ba4\u503c\uff1a1024\uff0c\u5f53\u4e34\u65f6\u6587\u4ef6\u8fbe\u5230\u8be5\u5927\u5c0f\uff08\u5355\u4f4d\uff1abytes\uff09\u65f6\uff0c\u6eda\u52a8\u6210\u76ee\u6807\u6587\u4ef6\u3002\u5982\u679c\u8bbe\u7f6e\u62100\uff0c\u5219\u8868\u793a\u4e0d\u6839\u636e\u4e34\u65f6\u6587\u4ef6\u5927\u5c0f\u6765\u6eda\u52a8\u6587\u4ef6\u3002\n\nrollCount\n\u9ed8\u8ba4\u503c\uff1a10\uff0c\u5f53events\u6570\u636e\u8fbe\u5230\u8be5\u6570\u91cf\u65f6\u5019\uff0c\u5c06\u4e34\u65f6\u6587\u4ef6\u6eda\u52a8\u6210\u76ee\u6807\u6587\u4ef6\uff0c\u5982\u679c\u8bbe\u7f6e\u62100\uff0c\u5219\u8868\u793a\u4e0d\u6839\u636eevents\u6570\u636e\u6765\u6eda\u52a8\u6587\u4ef6\u3002\n<\/code><\/pre>\n<blockquote><p>\n  \u5229\u7528secondarynamenode\u624b\u52a8\u6062\u590dnamenode\n<\/p><\/blockquote>\n<p>\u901a\u8fc7\u89c2\u5bdfdatanode,namenode,secondary namenode\u4e0b\u7684VERSION\u8fd9\u4e2a\u6587\u4ef6\u4f60\u4f1a\u6ce8\u610f\u5230scondarynamenode\u7684\u4e0enamenode\u4e00\u81f4<\/p>\n<pre><code class=\"language-properties line-numbers\">#\/opt\/hadoop-2.7.7\/data\/dfs\/data\/current\n#Sun Apr 04 00:38:37 CST 2021\nstorageID=DS-50ab0fb0-0f16-4b66-8c33-eab6b8c3ea62\nclusterID=CID-9f9898aa-a9cf-4fc9-9aa9-4774e0393ded\ncTime=0\ndatanodeUuid=bb12d0e9-8929-4980-8f51-23a2ca482ae8\nstorageType=DATA_NODE\nlayoutVersion=-56\n\n\/opt\/hadoop-2.7.7\/data\/dfs\/name\/current\n#Mon Mar 29 16:52:02 CST 2021\nnamespaceID=562029888\nclusterID=CID-9f9898aa-a9cf-4fc9-9aa9-4774e0393ded\ncTime=0\nstorageType=NAME_NODE\nblockpoolID=BP-1164465073-192.168.236.236-1598873871309\nlayoutVersion=-63\n\n\/opt\/hadoop-2.7.7\/data\/dfs\/namesecondary\/current\n#Sun Apr 04 02:40:16 CST 2021\nnamespaceID=562029888\nclusterID=CID-9f9898aa-a9cf-4fc9-9aa9-4774e0393ded\ncTime=0\nstorageType=NAME_NODE\nblockpoolID=BP-1164465073-192.168.236.236-1598873871309\nlayoutVersion=-63\n<\/code><\/pre>\n<p>\u628anamenodesecondary\u62f7\u8d1d\u8fd9\u4e2adfs\u6587\u4ef6\u5939\u4e0b\u91cd\u547d\u540d\u4e3aname\u5373\u53ef<\/p>\n<ul>\n<li>\u5982\u679c\u4e00\u53f0\u673a\u5668\u7684datanode\u6302\u6389\u4e4b\u540e\u6216\u8005\u8bf4\u662f\u8981\u52a0\u5165\u8fdb\u6765<\/li>\n<\/ul>\n<pre><code class=\"language-bash line-numbers\">\u6267\u884c\u4e0b\u9762\u8fd9\u4e2a\u547d\u4ee4\u524d\uff0c\u628adatanode\u7684VERSION\u7684clusterID\u7684\u503c\u6362\u6210\u60f3\u8981\u52a0\u5165\u7684\u90a3\u4e2a\u96c6\u7fa4\u7684clusterID\nhadoop-daemon.sh start datanode   \n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>\u5b98\u7f51\u7ed9\u51fa\u7684\u4ecb\u7ecd The Hadoop Distributed File System (HDFS) is a  [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[7],"tags":[],"_links":{"self":[{"href":"http:\/\/www.specialwu.com\/index.php?rest_route=\/wp\/v2\/posts\/1634"}],"collection":[{"href":"http:\/\/www.specialwu.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.specialwu.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.specialwu.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.specialwu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1634"}],"version-history":[{"count":16,"href":"http:\/\/www.specialwu.com\/index.php?rest_route=\/wp\/v2\/posts\/1634\/revisions"}],"predecessor-version":[{"id":1928,"href":"http:\/\/www.specialwu.com\/index.php?rest_route=\/wp\/v2\/posts\/1634\/revisions\/1928"}],"wp:attachment":[{"href":"http:\/\/www.specialwu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1634"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.specialwu.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1634"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.specialwu.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}